Nov 03 13:08:02.269338 ip-10-0-139-248 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 13:08:02.695244 ip-10-0-139-248 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:08:02.695244 ip-10-0-139-248 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 13:08:02.695244 ip-10-0-139-248 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:08:02.695244 ip-10-0-139-248 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:08:02.698375 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698296    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:02.698375 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698375    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:02.698375 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698379    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698383    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698387    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698391    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698395    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698398    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698400    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698403    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698406    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698408    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698411    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698414    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698417    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698419    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698422    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698425    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698427    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698430    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698433    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698435    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:02.698465 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698438    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698442    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698446    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698449    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698452    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698454    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698457    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698459    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698462    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698465    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698467    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698470    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698473    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698476    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698480    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698482    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698485    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698488    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698490    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:02.698929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698492    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698495    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698498    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698500    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698503    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698507    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698510    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698513    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698516    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698519    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698522    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698524    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698527    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698529    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698540    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698543    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698546    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698549    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698551    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:02.699431 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698554    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698557    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698559    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698562    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698565    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698568    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698570    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698573    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698576    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698579    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698581    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698584    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698587    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698589    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698592    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698595    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698597    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698600    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698602    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698604    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:02.699884 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698607    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698609    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698612    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698615    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698618    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698620    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698623    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698626    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698629    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698631    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698634    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698636    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698639    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698641    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698644    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698646    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698649    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698651    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698654    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:02.700673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698656    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698659    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698662    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698665    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698668    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698670    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.698673    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701212    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701230    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701236    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701240    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701246    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701250    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701255    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701259    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701264    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701268    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701272    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701276    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701280    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:02.701528 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701285    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701289    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701294    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701298    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701302    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701306    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701310    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701314    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701318    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701323    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701328    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701332    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701336    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701340    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701344    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701348    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701353    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701357    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701362    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701368    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:02.702401 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701373    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701378    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701382    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701387    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701391    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701395    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701400    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701404    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701408    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701412    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701417    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701422    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701426    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701430    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701437    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701444    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701451    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701459    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701464    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:02.703264 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701469    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701473    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701477    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701481    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701485    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701493    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701498    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701502    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701506    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701511    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701516    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701520    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701524    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701529    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701533    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701536    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701542    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701546    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701550    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701555    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:02.703729 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701559    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701563    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701567    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701571    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701576    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701580    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701584    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701588    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701593    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701597    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701602    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701607    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701611    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701615    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701620    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701624    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701628    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701633    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701637    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:02.704381 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701641    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701645    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701649    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701653    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701658    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701663    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701667    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701671    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701676    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701680    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701685    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701689    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701693    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701699    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.701703    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702571    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702580    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702585    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702589    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:02.705189 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702593    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702598    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702602    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702607    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702611    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702616    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702620    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702625    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702629    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702633    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702638    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702644    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702649    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702653    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702657    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702662    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702666    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702670    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702675    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702679    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:02.706582 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702683    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702687    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702692    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702696    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702700    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702705    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702709    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702713    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702718    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702722    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702726    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702730    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702734    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702739    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702743    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702747    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702751    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702755    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702765    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702769    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:02.707318 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702774    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702779    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702783    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702788    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702792    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702796    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702800    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702804    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702809    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702813    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702817    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702821    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702826    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702830    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702834    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702839    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702843    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702847    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702851    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:02.707788 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702855    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702860    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702864    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702868    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702872    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702876    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702880    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702884    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702889    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702893    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702897    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702902    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702907    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702911    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702915    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702920    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702924    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702928    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702933    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:02.708329 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702939    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702943    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702947    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702951    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702955    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702959    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702964    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702968    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702972    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702976    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702981    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702988    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702992    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.702997    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703001    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703005    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703009    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703013    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703017    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703021    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:02.708820 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703046    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703056    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703063    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.703069    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704066    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704081    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704093    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704100    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704109    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704114    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704120    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704127    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704132    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704137    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704143    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704149    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704154    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704159    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704163    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704168    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704173    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704177    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704185    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 03 13:08:02.709327 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704192    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704198    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704203    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704208    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704215    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704220    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704226    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704231    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704236    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704241    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704245    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704251    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704255    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704262    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704268    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704272    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704278    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704283    2576 flags.go:64] FLAG: --enable-server="true"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704287    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704295    2576 flags.go:64] FLAG: --event-burst="100"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704299    2576 flags.go:64] FLAG: --event-qps="50"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704304    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704310    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704315    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 03 13:08:02.709862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704321    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704326    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704331    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704336    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704340    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704346    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704350    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704355    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704362    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704371    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704376    2576 flags.go:64] FLAG: --feature-gates=""
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704388    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704393    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704398    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704403    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704410    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704415    2576 flags.go:64] FLAG: --help="false"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704421    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704427    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704432    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704437    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704443    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704448    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 13:08:02.710521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704453    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704458    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704464    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704469    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704474    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704479    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704484    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704489    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704493    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704498    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704503    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704508    2576 flags.go:64] FLAG: --lock-file=""
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704513    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704518    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704522    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704532    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704536    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704543    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704548    2576 flags.go:64] FLAG: --logging-format="text"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704553    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704558    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704563    2576 flags.go:64] FLAG: --manifest-url=""
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704568    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704575    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704581    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 13:08:02.711090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704588    2576 flags.go:64] FLAG: --max-pods="110"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704592    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704597    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704602    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704607    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704612    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704616    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704621    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704633    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704639    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704644    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704649    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704654    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704659    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704664    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704669    2576 flags.go:64] FLAG: --port="10250"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704673    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704678    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0be33d50c0802780b"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704683    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704688    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704693    2576 flags.go:64] FLAG: --register-node="true"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704697    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704704    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704709    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704715    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704719    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 03 13:08:02.711737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704725    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704730    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704735    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704740    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704745    2576 flags.go:64] FLAG: --runonce="false"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704754    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704759    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704765    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704769    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704774    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704779    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704783    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704788    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704793    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704798    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704802    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704807    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704813    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704818    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704822    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704831    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704836    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704841    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704848    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704853    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 13:08:02.712407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704857    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704861    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704866    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704871    2576 flags.go:64] FLAG: --v="2"
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704878    2576 flags.go:64] FLAG: --version="false"
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704886    2576 flags.go:64] FLAG: --vmodule=""
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704892    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.704898    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705154    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705162    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705167    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705172    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705178    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705182    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705187    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705192    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705196    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705200    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705204    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705208    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705212    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:02.712988 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705216    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705223    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705230    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705236    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705241    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705246    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705250    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705255    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705259    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705263    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705268    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705272    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705276    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705280    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705285    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705289    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705295    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705299    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705304    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:02.713510 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705309    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705313    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705318    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705322    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705329    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705333    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705338    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705342    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705346    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705351    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705355    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705360    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705364    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705368    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705372    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705377    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705381    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705385    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705391    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:02.713963 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705395    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705400    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705405    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705409    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705413    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705417    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705422    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705426    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705431    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705435    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705442    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705446    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705450    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705455    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705459    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705463    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705467    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705473    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705478    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705482    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:02.714463 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705486    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705491    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705495    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705499    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705503    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705508    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705512    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705517    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705522    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705526    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705530    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705535    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705540    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705544    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705548    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705552    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705557    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705561    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705567    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705573    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:02.714937 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705578    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705582    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705589    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705593    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705598    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705602    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705605    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705608    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705611    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705616    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705619    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705622    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705624    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705627    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705630    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.705663    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:08:02.715500 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705710    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705714    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705719    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705723    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705726    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705729    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705732    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705734    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705737    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705741    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705743    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705746    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705748    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705751    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705754    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705756    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705759    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705762    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705766    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705768    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:02.715896 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705771    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705773    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705776    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705779    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705782    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705784    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705789    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705792    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705794    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705797    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705799    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705802    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705805    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705809    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705812    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705816    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705818    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705821    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705824    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:02.716402 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705827    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705829    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705832    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705835    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705837    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705840    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705842    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705845    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705847    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705850    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705852    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705856    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705859    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705863    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705865    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705868    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705871    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705874    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705876    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:02.716852 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705880    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705883    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705886    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705888    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705891    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705893    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705896    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705899    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705901    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705904    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705906    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705909    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705911    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705914    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705916    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705919    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705921    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705924    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705928    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:02.717326 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705930    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705933    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705935    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705938    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705940    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705944    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705947    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705949    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705952    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705954    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705957    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705960    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705962    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705966    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705968    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705971    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705974    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705976    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705979    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705981    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:02.717809 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705984    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705986    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705989    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705991    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705995    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.705998    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.706001    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.706003    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.706006    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.713120    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.713136    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.713153    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.713159    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713237    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713243    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713246    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713250    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713253    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713255    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713258    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:02.718343 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713261    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713263    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713266    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713269    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713271    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713274    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713277    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713279    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713282    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713285    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713287    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713290    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713293    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713295    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713298    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713301    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713303    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713306    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713308    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:02.718818 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713311    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713314    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713316    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713320    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713323    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713326    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713329    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713331    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713334    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713337    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713340    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713342    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713345    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713347    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713350    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713352    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713355    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713357    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713360    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713362    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:02.719325 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713365    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713368    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713371    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713373    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713376    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713378    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713381    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713384    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713386    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713388    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713391    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713393    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713396    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713399    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713402    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713405    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713408    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713411    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713414    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:02.719782 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713416    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713419    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713423    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713427    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713430    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713433    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713436    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713439    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713441    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713444    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713446    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713449    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713452    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713454    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713457    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713459    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713462    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713464    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713467    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:02.720259 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713469    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713472    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713475    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713477    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713480    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713482    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713485    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713488    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713491    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713494    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713497    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713500    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713503    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713506    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713508    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713512    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713516    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713519    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713522    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713525    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:02.720694 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713528    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713530    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.713559    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713710    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713714    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713717    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713720    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713723    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713726    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713728    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713731    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713734    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713736    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713740    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713744    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713748    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:02.721181 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713751    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713754    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713757    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713759    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713762    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713765    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713768    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713771    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713774    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713777    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713780    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713782    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713785    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713787    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713790    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713793    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713795    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713798    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713800    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713803    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:02.721556 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713806    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713809    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713811    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713814    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713816    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713819    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713822    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713825    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713829    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713832    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713835    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713838    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713840    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713843    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713845    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713848    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713850    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713853    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713855    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:02.722013 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713858    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713860    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713863    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713866    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713868    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713871    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713873    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713876    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713878    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713881    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713884    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713886    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713889    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713891    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713894    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713896    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713899    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713901    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713903    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:02.722535 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713906    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713909    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713911    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713914    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713917    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713920    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713922    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713925    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713927    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713929    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713932    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713935    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713937    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713940    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713942    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713945    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713947    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713950    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713952    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:02.722969 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713955    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713958    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713960    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713962    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713965    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713968    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713970    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713973    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713976    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713978    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713981    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713984    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713986    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713989    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713991    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.713994    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:02.723429 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.714017    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714068    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714073    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714076    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714078    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714081    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714083    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714086    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714089    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714092    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714094    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714097    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714100    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714103    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714106    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714109    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714111    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714115    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714119    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:08:02.723807 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714122    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714124    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714127    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714129    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714132    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714134    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714137    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714139    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714142    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714144    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714147    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714149    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714152    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714154    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714157    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714160    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714162    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714165    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714168    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714170    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:08:02.724291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714173    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714175    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714178    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714181    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714183    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714185    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714188    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714193    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714196    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714199    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714202    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714204    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714207    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714210    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714212    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714215    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714217    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714220    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714222    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:08:02.724929 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714225    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714228    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714230    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714232    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714235    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714237    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714240    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714243    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714245    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714248    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714250    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714253    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714255    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714258    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714260    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714263    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714265    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714268    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714270    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:08:02.725673 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714273    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714277    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714280    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714283    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714285    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714288    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714290    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714293    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714295    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714298    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714300    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714302    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714305    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714307    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714310    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714312    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714316    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714320    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714324    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714327    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:08:02.726294 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714329    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714332    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714334    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714337    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714340    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714343    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714345    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714348    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714350    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:02.714353    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.714410    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.716359    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.717284    2576 server.go:1014] "Starting client certificate rotation"
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.717382    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.718304    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.719435    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 13:08:02.726776 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.725971    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 13:08:02.737841 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.737826    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 03 13:08:02.740352 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.740337    2576 log.go:25] "Validated CRI v1 image API"
Nov 03 13:08:02.741142 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.741120    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 13:08:02.747216 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.747195    2576 fs.go:135] Filesystem UUIDs: map[3fdab71c-10fc-409f-9c55-acb6cf004c6e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dcffbb01-f8e8-45d6-b39f-765a512536f5:/dev/nvme0n1p3]
Nov 03 13:08:02.747280 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.747214    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 13:08:02.747917 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.747897    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 13:08:02.751609 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.751492    2576 manager.go:217] Machine: {Timestamp:2026-11-03 13:08:02.7510114 +0000 UTC m=+0.358628813 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097948 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2601d6dc49d09845c1e148b68030d4 SystemUUID:ec2601d6-dc49-d098-45c1-e148b68030d4 BootID:c74f5349-78af-46dc-a695-4c70919b0570 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:52:1f:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:52:1f:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:80:0d:1a:f9:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 13:08:02.751609 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.751594    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 13:08:02.751758 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.751696    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 13:08:02.756483 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.756453    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 13:08:02.756624 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.756485    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-248.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 13:08:02.756669 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.756627    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 13:08:02.756669 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.756635    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 13:08:02.756718 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.756669    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 13:08:02.756718 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.756686    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 13:08:02.757519 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.757509    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 13:08:02.761260 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.761248    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 13:08:02.761388 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.761379    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 13:08:02.761430 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.761410    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 13:08:02.761430 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.761421    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 13:08:02.761490 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.761431    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 13:08:02.761490 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.761440    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 03 13:08:02.761490 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.761449    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 13:08:02.761814 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.761803    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 13:08:02.764359 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.764340    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 13:08:02.769734 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.769721    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 13:08:02.770506 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.770489    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 13:08:02.770561 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.770523    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 13:08:02.770699 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.770675    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 13:08:02.770764 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.770705    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 13:08:02.770764 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.770716    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 13:08:02.770764 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.770725    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 13:08:02.770764 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.770733    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 13:08:02.770764 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.770741    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 13:08:02.771976 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.771953    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 13:08:02.771976 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.771977    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 13:08:02.772136 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.771996    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 13:08:02.772136 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.772004    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 13:08:02.772136 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.772014    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 13:08:02.772136 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.772025    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 13:08:02.772768 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.772733    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 13:08:02.772859 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.772850    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 13:08:02.773604 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.772843    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-92z6n"
Nov 03 13:08:02.777424 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.777409    2576 server.go:1263] "Started kubelet"
Nov 03 13:08:02.777540 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.777501    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 13:08:02.777540 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.777490    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 13:08:02.777619 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.777562    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 13:08:02.777950 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.777930    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-92z6n"
Nov 03 13:08:02.778167 ip-10-0-139-248 systemd[1]: Started Kubernetes Kubelet.
Nov 03 13:08:02.778754 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.778734    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 13:08:02.781319 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.781304    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 13:08:02.784629 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.784607    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-248.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 13:08:02.784777 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.784755    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 13:08:02.784863 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.784757    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-248.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 13:08:02.785169 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.784151    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-248.us-east-2.compute.internal.18e48ae87f8de488  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-248.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-248.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 13:08:02.777351304 +0000 UTC m=+0.384968717,LastTimestamp:2026-11-03 13:08:02.777351304 +0000 UTC m=+0.384968717,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-248.us-east-2.compute.internal,}"
Nov 03 13:08:02.785251 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.785204    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 13:08:02.785251 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.785233    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 13:08:02.785683 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.785656    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.786204    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.786597    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787154    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787176    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787220    2576 factory.go:55] Registering systemd factory
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787237    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787315    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787325    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787503    2576 factory.go:153] Registering CRI-O factory
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787522    2576 factory.go:223] Registration of the crio container factory successfully
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787621    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787636    2576 factory.go:103] Registering Raw factory
Nov 03 13:08:02.788131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.787649    2576 manager.go:1199] Started watching for new ooms in manager
Nov 03 13:08:02.788660 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.788582    2576 manager.go:319] Starting recovery of all containers
Nov 03 13:08:02.790929 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.790910    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:02.792107 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.792080    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-248.us-east-2.compute.internal\" not found" node="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:02.794238 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.794211    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 13:08:02.804142 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.804123    2576 manager.go:324] Recovery completed
Nov 03 13:08:02.805933 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.805861    2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 03 13:08:02.808991 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.808973    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:08:02.810887 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.810870    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:08:02.810983 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.810907    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:08:02.810983 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.810924    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:08:02.812230 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.812213    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 03 13:08:02.812230 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.812227    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 13:08:02.812363 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.812247    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 13:08:02.812363 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.812218    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:08:02.813161 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.813137    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:08:02.813251 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.813180    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:08:02.813251 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.813197    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:08:02.814443 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.814431    2576 policy_none.go:50] "Start"
Nov 03 13:08:02.814498 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.814447    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 13:08:02.814498 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.814457    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 13:08:02.816513 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.816501    2576 policy_none.go:44] "Start"
Nov 03 13:08:02.842771 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.842738    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 13:08:02.847090 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.847065    2576 manager.go:341] "Starting Device Plugin manager"
Nov 03 13:08:02.847193 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.847101    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 13:08:02.847193 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.847110    2576 server.go:87] "Starting device plugin registration server"
Nov 03 13:08:02.847346 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.847334    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 13:08:02.847406 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.847346    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 13:08:02.847456 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.847445    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 13:08:02.847531 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.847517    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 13:08:02.847531 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.847528    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 13:08:02.848156 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.848134    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 13:08:02.848246 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.848189    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:02.921612 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.921585    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 13:08:02.921612 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.921611    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 13:08:02.923065 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.921634    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 13:08:02.923065 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.921665    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 13:08:02.924223 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.924205    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:02.947684 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.947572    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:08:02.949634 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.949618    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:08:02.949730 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.949648    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:08:02.949730 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.949659    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:08:02.949730 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.949684    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:02.957653 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:02.957637    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:02.957736 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.957661    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-248.us-east-2.compute.internal\": node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:02.973187 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:02.973165    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:03.022195 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.022148    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"]
Nov 03 13:08:03.022360 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.022228    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:08:03.023226 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.023201    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:08:03.023298 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.023234    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:08:03.023298 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.023243    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:08:03.024881 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.024868    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:08:03.025053 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.025025    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.025091 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.025069    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:08:03.025609 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.025593    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:08:03.025688 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.025629    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:08:03.025688 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.025642    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:08:03.025688 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.025658    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:08:03.025688 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.025676    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:08:03.025817 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.025691    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:08:03.027099 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.027086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.027160 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.027109    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:08:03.027760 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.027746    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:08:03.027833 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.027772    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:08:03.027833 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.027786    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:08:03.054559 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:03.054538    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found" node="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.056663 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:03.056646    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found" node="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.074165 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:03.074147    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:03.089532 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.089509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bebcb79842815c8633e47e5468f5a5a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"1bebcb79842815c8633e47e5468f5a5a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.089620 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.089535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.089620 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.089553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.174604 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:03.174577    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:03.189938 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.189914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bebcb79842815c8633e47e5468f5a5a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"1bebcb79842815c8633e47e5468f5a5a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.190067 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.189948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.190067 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.189970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.190067 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.190007    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.190067 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.190020    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.190067 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.190008    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bebcb79842815c8633e47e5468f5a5a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"1bebcb79842815c8633e47e5468f5a5a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.275400 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:03.275330    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:03.358858 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.358821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.360968 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:03.360950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:03.376449 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:03.376308    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:03.476898 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:03.476828    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:04.909611 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:04.909522    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-248.us-east-2.compute.internal\" not found"
Nov 03 13:08:04.934556 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:04.934531    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:05.019745 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.019716    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:05.029818 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.029796    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:08:05.033774 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.033753    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:08:05.043714 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.043691    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:08:05.049817 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.049796    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 13:08:05.049951 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.049929    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 13:08:05.049990 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.049969    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 13:08:05.050030 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.049967    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 13:08:05.094126 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.094085    2576 apiserver.go:52] "Watching apiserver"
Nov 03 13:08:05.118364 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.118335    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 13:08:05.118504 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.118339    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 13:03:04 +0000 UTC" deadline="2028-08-19 18:00:15.161765236 +0000 UTC"
Nov 03 13:08:05.118504 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.118385    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15724h52m10.043383604s"
Nov 03 13:08:05.126689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.126664    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:05.129505 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.129488    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 13:08:05.139878 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.139855    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 13:08:05.140259 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.140243    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-72hxg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal","openshift-multus/multus-4bn8t","openshift-multus/multus-additional-cni-plugins-k445k","openshift-ovn-kubernetes/ovnkube-node-k4pjp","kube-system/global-pull-secret-syncer-lr88l","kube-system/konnectivity-agent-729bm","openshift-dns/node-resolver-lx28x","openshift-network-diagnostics/network-check-target-dnqj9","kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-5gj6w","openshift-image-registry/node-ca-xptcl","openshift-network-operator/iptables-alerter-zkrzz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"]
Nov 03 13:08:05.142446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.142429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:05.145527 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.145508    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.146524 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.146506    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.150126 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150087    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c596b\""
Nov 03 13:08:05.150230 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150093    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vwt8x\""
Nov 03 13:08:05.150285 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150263    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-62cxl\""
Nov 03 13:08:05.150457 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150443    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 13:08:05.150512 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150473    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 13:08:05.150512 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150481    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 13:08:05.150512 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150499    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 13:08:05.150512 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150483    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.150674 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150547    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 13:08:05.150674 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150549    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 13:08:05.150674 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150574    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:08:05.150674 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150579    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 13:08:05.150674 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.150578    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 13:08:05.151113 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.151099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.153781 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.153760    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.153874 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.153828    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:05.155376 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.155357    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ftr7d"
Nov 03 13:08:05.155644 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.155627    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wv2nv\""
Nov 03 13:08:05.155795 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.155783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:05.155925 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.155908    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:05.157899 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.157882    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.160072 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.160032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.162196 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.162173    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.164496 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.164478    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:05.164574 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.164556    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:05.166649 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.166635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.167673 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.167655    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ftr7d"
Nov 03 13:08:05.219738 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.219714    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 13:08:05.226719 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.226694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-netns\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.226839 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.226727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-conf-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.226839 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.226745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrxp\" (UniqueName: \"kubernetes.io/projected/0d2aa382-9410-48f2-b139-3d9d12a26b3b-kube-api-access-rfrxp\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:05.226839 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.226763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22aec86e-8e32-4e9b-82d7-218a2ceb3926-agent-certs\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:05.226947 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.226832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.226947 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.226875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48g5\" (UniqueName: \"kubernetes.io/projected/3acab04e-53c1-4ceb-b571-246d23c55934-kube-api-access-j48g5\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.226947 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.226919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.227055 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.226966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cni-binary-copy\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.227055 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-multus-certs\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.227122 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-systemd-units\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.227122 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-config\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.227179 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-host\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.227179 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-sys-fs\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.227179 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.227299 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-socket-dir-parent\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.227299 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22aec86e-8e32-4e9b-82d7-218a2ceb3926-konnectivity-ca\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:05.227299 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-sys\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.227299 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227292    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-system-cni-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.227424 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.227424 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a22ec490-f623-4cda-a693-e6669a1a3324-ovn-node-metrics-cert\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.227424 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-tmp\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.227517 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-daemon-config\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.227517 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-etc-kubernetes\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.227517 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cnibin\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.227517 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-log-socket\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.227639 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.227639 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227554    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-registration-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.227639 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pvl\" (UniqueName: \"kubernetes.io/projected/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-kube-api-access-46pvl\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.227639 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-bin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.227778 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.227778 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94tq\" (UniqueName: \"kubernetes.io/projected/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-kube-api-access-d94tq\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.227778 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-script-lib\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.227778 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-hostroot\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.227778 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-dbus\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.227778 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfr92\" (UniqueName: \"kubernetes.io/projected/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-kube-api-access-xfr92\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.227991 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.227991 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-kubelet\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.227991 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-ovn\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.227991 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bt2\" (UniqueName: \"kubernetes.io/projected/a22ec490-f623-4cda-a693-e6669a1a3324-kube-api-access-t9bt2\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.227991 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.227961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-conf\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.228150 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:05.228150 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228049    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-serviceca\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.228150 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-slash\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228150 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-netd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228150 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-node-log\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228357 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-env-overrides\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228357 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:05.228357 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8db0e985-d36d-485d-91ae-917f751a61a1-host-slash\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.228357 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228247    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cnibin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.228357 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysconfig\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.228357 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-etc-selinux\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.228544 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-etc-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228544 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228379    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-bin\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228544 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-lib-modules\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.228544 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8db0e985-d36d-485d-91ae-917f751a61a1-iptables-alerter-script\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.228544 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-host\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.228544 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-systemd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228738 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7xv\" (UniqueName: \"kubernetes.io/projected/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kube-api-access-4g7xv\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.228738 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-multus\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.228738 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfm9p\" (UniqueName: \"kubernetes.io/projected/aa3f9f6e-77c4-4053-9289-65469d56d6ac-kube-api-access-zfm9p\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.228738 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.228738 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-netns\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228738 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-modprobe-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.228738 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-systemd\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.228963 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-system-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.228963 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228794    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-kubelet\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.228963 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-kubelet-config\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.228963 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-binary-copy\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.228963 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-var-lib-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228963 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.228963 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-socket-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.229173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-hosts-file\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.229173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.228989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-k8s-cni-cncf-io\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.229173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.229005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.229173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.229039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-run\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.229173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.229064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-var-lib-kubelet\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.229173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.229089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-etc-tuned\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.229173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.229124    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-device-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.229173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.229147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-tmp-dir\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.229173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.229171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvx7\" (UniqueName: \"kubernetes.io/projected/8db0e985-d36d-485d-91ae-917f751a61a1-kube-api-access-jvvx7\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.229477 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.229196    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-kubernetes\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.254058 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254032    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:08:05.254267 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254195    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 13:08:05.254267 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254198    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 13:08:05.254267 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254204    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 13:08:05.254267 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254242    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 13:08:05.254458 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254344    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 13:08:05.254458 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254354    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 13:08:05.254458 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254375    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 13:08:05.254458 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254415    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 13:08:05.254648 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254612    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 13:08:05.254702 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254653    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s4cl2\""
Nov 03 13:08:05.254702 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254675    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 13:08:05.254780 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254748    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 13:08:05.254926 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254912    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 13:08:05.254977 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254941    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 13:08:05.255022 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.254990    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 13:08:05.255083 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.255042    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-knm89\""
Nov 03 13:08:05.255083 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.255049    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 13:08:05.255083 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.255053    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 13:08:05.255083 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.255064    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mrd6\""
Nov 03 13:08:05.255230 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.255156    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-psqgf\""
Nov 03 13:08:05.255230 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.255160    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jqlk5\""
Nov 03 13:08:05.255305 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.255265    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 13:08:05.329774 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-netns\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.329774 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-conf-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329793    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrxp\" (UniqueName: \"kubernetes.io/projected/0d2aa382-9410-48f2-b139-3d9d12a26b3b-kube-api-access-rfrxp\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22aec86e-8e32-4e9b-82d7-218a2ceb3926-agent-certs\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329875    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-netns\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329875    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-conf-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329934    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329938    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48g5\" (UniqueName: \"kubernetes.io/projected/3acab04e-53c1-4ceb-b571-246d23c55934-kube-api-access-j48g5\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.329990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cni-binary-copy\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-multus-certs\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.330054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-systemd-units\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-config\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330083    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-host\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330104    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-systemd-units\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-sys-fs\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330105    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-multus-certs\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-host\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330154    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-socket-dir-parent\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330194    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-sys-fs\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22aec86e-8e32-4e9b-82d7-218a2ceb3926-konnectivity-ca\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-sys\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-system-cni-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a22ec490-f623-4cda-a693-e6669a1a3324-ovn-node-metrics-cert\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330309    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.330605 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330314    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-socket-dir-parent\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-tmp\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330362    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-system-cni-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330370    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-sys\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-daemon-config\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-etc-kubernetes\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330507    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330545    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-etc-kubernetes\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cnibin\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-log-socket\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-registration-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pvl\" (UniqueName: \"kubernetes.io/projected/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-kube-api-access-46pvl\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330714    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-config\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-bin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330722    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cni-binary-copy\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330748    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22aec86e-8e32-4e9b-82d7-218a2ceb3926-konnectivity-ca\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:05.331448 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330806    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cnibin\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330805    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-registration-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94tq\" (UniqueName: \"kubernetes.io/projected/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-kube-api-access-d94tq\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-script-lib\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330846    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-bin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330881    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-daemon-config\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-hostroot\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.330995    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-dbus\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331029    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-log-socket\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfr92\" (UniqueName: \"kubernetes.io/projected/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-kube-api-access-xfr92\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.331089    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331170    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-hostroot\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.331176    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret podName:2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df nodeName:}" failed. No retries permitted until 2026-11-03 13:08:05.831161526 +0000 UTC m=+2.106597616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret") pod "global-pull-secret-syncer-lr88l" (UID: "2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-kubelet\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332224 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-ovn\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331297    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-kubelet\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bt2\" (UniqueName: \"kubernetes.io/projected/a22ec490-f623-4cda-a693-e6669a1a3324-kube-api-access-t9bt2\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-conf\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331369    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-dbus\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331393    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-ovn\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331435    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-serviceca\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-slash\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331472    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-conf\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-netd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331497    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-netd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-node-log\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331534    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-slash\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331560    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-node-log\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-env-overrides\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.331631    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:05.332902 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8db0e985-d36d-485d-91ae-917f751a61a1-host-slash\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cnibin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysconfig\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-etc-selinux\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-etc-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-bin\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331826    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-lib-modules\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8db0e985-d36d-485d-91ae-917f751a61a1-iptables-alerter-script\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-host\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-systemd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331950    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-script-lib\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.331989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7xv\" (UniqueName: \"kubernetes.io/projected/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kube-api-access-4g7xv\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332020    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-multus\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cnibin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332042    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-env-overrides\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332066    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-systemd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfm9p\" (UniqueName: \"kubernetes.io/projected/aa3f9f6e-77c4-4053-9289-65469d56d6ac-kube-api-access-zfm9p\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.333689 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332098    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332119    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-etc-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-netns\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332143    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-etc-selinux\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332161    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-host\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysconfig\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332196    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-bin\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-multus\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332254    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8db0e985-d36d-485d-91ae-917f751a61a1-host-slash\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.332328    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs podName:0d2aa382-9410-48f2-b139-3d9d12a26b3b nodeName:}" failed. No retries permitted until 2026-11-03 13:08:05.83231335 +0000 UTC m=+2.107749429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs") pod "network-metrics-daemon-5gj6w" (UID: "0d2aa382-9410-48f2-b139-3d9d12a26b3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-modprobe-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-systemd\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-system-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332423    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-lib-modules\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-kubelet\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332484    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-systemd\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.334613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-kubelet-config\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-binary-copy\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332571    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-modprobe-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-var-lib-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332649    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-var-lib-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332662    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-kubelet\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-socket-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-hosts-file\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-k8s-cni-cncf-io\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-run\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-var-lib-kubelet\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-etc-tuned\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-device-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.332977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-tmp-dir\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvx7\" (UniqueName: \"kubernetes.io/projected/8db0e985-d36d-485d-91ae-917f751a61a1-kube-api-access-jvvx7\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.335446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-kubernetes\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333032    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-socket-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-hosts-file\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-var-lib-kubelet\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333149    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-kubernetes\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333178    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-system-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333199    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-k8s-cni-cncf-io\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-serviceca\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333283    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-device-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333473    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8db0e985-d36d-485d-91ae-917f751a61a1-iptables-alerter-script\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333568    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-tmp-dir\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333591    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-netns\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333665    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-kubelet-config\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-run\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.333981    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-tmp\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.334065    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-binary-copy\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.336239 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.334128    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22aec86e-8e32-4e9b-82d7-218a2ceb3926-agent-certs\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:05.337092 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.334177    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.337092 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.334186    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a22ec490-f623-4cda-a693-e6669a1a3324-ovn-node-metrics-cert\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.337092 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.335942    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-etc-tuned\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.337791 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.337767    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48g5\" (UniqueName: \"kubernetes.io/projected/3acab04e-53c1-4ceb-b571-246d23c55934-kube-api-access-j48g5\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.338459 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.338428    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrxp\" (UniqueName: \"kubernetes.io/projected/0d2aa382-9410-48f2-b139-3d9d12a26b3b-kube-api-access-rfrxp\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:05.341131 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.341109    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pvl\" (UniqueName: \"kubernetes.io/projected/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-kube-api-access-46pvl\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.342853 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.342596    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:08:05.342853 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.342602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfr92\" (UniqueName: \"kubernetes.io/projected/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-kube-api-access-xfr92\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.342853 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.342617    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:08:05.342853 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.342658    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ww4nl for pod openshift-network-diagnostics/network-check-target-dnqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:05.342853 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.342730    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl podName:09bd042b-b415-46cc-a7af-aa4a57ed11d5 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:05.842695675 +0000 UTC m=+2.118131765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ww4nl" (UniqueName: "kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl") pod "network-check-target-dnqj9" (UID: "09bd042b-b415-46cc-a7af-aa4a57ed11d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:05.343408 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.343386    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94tq\" (UniqueName: \"kubernetes.io/projected/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-kube-api-access-d94tq\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.344886 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.344868    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7xv\" (UniqueName: \"kubernetes.io/projected/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kube-api-access-4g7xv\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.344971 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.344929    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvx7\" (UniqueName: \"kubernetes.io/projected/8db0e985-d36d-485d-91ae-917f751a61a1-kube-api-access-jvvx7\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.345141 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.345127    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfm9p\" (UniqueName: \"kubernetes.io/projected/aa3f9f6e-77c4-4053-9289-65469d56d6ac-kube-api-access-zfm9p\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.345862 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.345833    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bt2\" (UniqueName: \"kubernetes.io/projected/a22ec490-f623-4cda-a693-e6669a1a3324-kube-api-access-t9bt2\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.394129 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.394081    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bebcb79842815c8633e47e5468f5a5a.slice/crio-5dc2b06e68791904bccd9bd8df8e3adf716a720cd98d047704d55b78877e53e3 WatchSource:0}: Error finding container 5dc2b06e68791904bccd9bd8df8e3adf716a720cd98d047704d55b78877e53e3: Status 404 returned error can't find the container with id 5dc2b06e68791904bccd9bd8df8e3adf716a720cd98d047704d55b78877e53e3
Nov 03 13:08:05.394502 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.394482    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe41ec8017368ca412baa7a38539e26d.slice/crio-24900b96e5aa02946a235af3656ad6fe62d6cad2008ca8569568f6034cd5d130 WatchSource:0}: Error finding container 24900b96e5aa02946a235af3656ad6fe62d6cad2008ca8569568f6034cd5d130: Status 404 returned error can't find the container with id 24900b96e5aa02946a235af3656ad6fe62d6cad2008ca8569568f6034cd5d130
Nov 03 13:08:05.401592 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.401575    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:08:05.473611 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.473528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:05.484695 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.484664    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22aec86e_8e32_4e9b_82d7_218a2ceb3926.slice/crio-5b01e044c6f2cf78b6d00062244c51da4915a3e081f4ac5cd50863f59cb9b8c8 WatchSource:0}: Error finding container 5b01e044c6f2cf78b6d00062244c51da4915a3e081f4ac5cd50863f59cb9b8c8: Status 404 returned error can't find the container with id 5b01e044c6f2cf78b6d00062244c51da4915a3e081f4ac5cd50863f59cb9b8c8
Nov 03 13:08:05.486854 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.486833    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:08:05.494934 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.494908    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db0e985_d36d_485d_91ae_917f751a61a1.slice/crio-d3028bcd6fdccb591f9779c911708f5c164c1563ceb1607361d7731bd3840f05 WatchSource:0}: Error finding container d3028bcd6fdccb591f9779c911708f5c164c1563ceb1607361d7731bd3840f05: Status 404 returned error can't find the container with id d3028bcd6fdccb591f9779c911708f5c164c1563ceb1607361d7731bd3840f05
Nov 03 13:08:05.508870 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.508834    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4bn8t"
Nov 03 13:08:05.517267 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.517242    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3f9f6e_77c4_4053_9289_65469d56d6ac.slice/crio-51c9df65e99c8e774fe7562ff97c0be38c8d184bc9565eee4dad5b27368d9c61 WatchSource:0}: Error finding container 51c9df65e99c8e774fe7562ff97c0be38c8d184bc9565eee4dad5b27368d9c61: Status 404 returned error can't find the container with id 51c9df65e99c8e774fe7562ff97c0be38c8d184bc9565eee4dad5b27368d9c61
Nov 03 13:08:05.544333 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.544305    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:08:05.553328 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.553304    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1e9fa4_b3c4_4f21_a4cb_9d846a39f642.slice/crio-1eb9f33307b58153cf0ce3ab50bac2a1e7e5004fc33cb2a6d87f064107d73d2e WatchSource:0}: Error finding container 1eb9f33307b58153cf0ce3ab50bac2a1e7e5004fc33cb2a6d87f064107d73d2e: Status 404 returned error can't find the container with id 1eb9f33307b58153cf0ce3ab50bac2a1e7e5004fc33cb2a6d87f064107d73d2e
Nov 03 13:08:05.564545 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.564523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:05.572494 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.572475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:08:05.572648 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.572626    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22ec490_f623_4cda_a693_e6669a1a3324.slice/crio-279e8c90c9de5e2c3911a2aac1346cae7ffe731d3e564ccaf0cd915f068cf39a WatchSource:0}: Error finding container 279e8c90c9de5e2c3911a2aac1346cae7ffe731d3e564ccaf0cd915f068cf39a: Status 404 returned error can't find the container with id 279e8c90c9de5e2c3911a2aac1346cae7ffe731d3e564ccaf0cd915f068cf39a
Nov 03 13:08:05.579841 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.579820    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acab04e_53c1_4ceb_b571_246d23c55934.slice/crio-defda70f1e3cf052c5a629d93ad869aacee0508bc4d8db8ba51bd76c437f92f8 WatchSource:0}: Error finding container defda70f1e3cf052c5a629d93ad869aacee0508bc4d8db8ba51bd76c437f92f8: Status 404 returned error can't find the container with id defda70f1e3cf052c5a629d93ad869aacee0508bc4d8db8ba51bd76c437f92f8
Nov 03 13:08:05.592442 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.592424    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:08:05.601460 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.601438    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca216a1_acaf_4fe6_b833_4fc7e25ad427.slice/crio-3ad193723cf185eca2af116534b69a27edc90f9ec878f7f94aa0d133122c5847 WatchSource:0}: Error finding container 3ad193723cf185eca2af116534b69a27edc90f9ec878f7f94aa0d133122c5847: Status 404 returned error can't find the container with id 3ad193723cf185eca2af116534b69a27edc90f9ec878f7f94aa0d133122c5847
Nov 03 13:08:05.606732 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.606713    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:08:05.613426 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.613409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:08:05.615365 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.615341    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffdebd2_3a79_4eb9_baf4_87b91d78ac01.slice/crio-0a61ef30690bfe69177dbda8ce018d0180dca73085a4be9b5826c651d34a2ae4 WatchSource:0}: Error finding container 0a61ef30690bfe69177dbda8ce018d0180dca73085a4be9b5826c651d34a2ae4: Status 404 returned error can't find the container with id 0a61ef30690bfe69177dbda8ce018d0180dca73085a4be9b5826c651d34a2ae4
Nov 03 13:08:05.621648 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:08:05.621625    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ccec32_8bf2_4f47_8d30_abd99b513d1b.slice/crio-f950a5f26311d75c891f5a30bac42e0715df627a3c435b5aa890ff02f0f747c5 WatchSource:0}: Error finding container f950a5f26311d75c891f5a30bac42e0715df627a3c435b5aa890ff02f0f747c5: Status 404 returned error can't find the container with id f950a5f26311d75c891f5a30bac42e0715df627a3c435b5aa890ff02f0f747c5
Nov 03 13:08:05.836429 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.836352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:05.836429 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.836390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:05.836601 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.836506    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:05.836601 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.836561    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret podName:2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df nodeName:}" failed. No retries permitted until 2026-11-03 13:08:06.836548569 +0000 UTC m=+3.111984647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret") pod "global-pull-secret-syncer-lr88l" (UID: "2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:05.836601 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.836506    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:05.836696 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.836636    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs podName:0d2aa382-9410-48f2-b139-3d9d12a26b3b nodeName:}" failed. No retries permitted until 2026-11-03 13:08:06.836625049 +0000 UTC m=+3.112061137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs") pod "network-metrics-daemon-5gj6w" (UID: "0d2aa382-9410-48f2-b139-3d9d12a26b3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:05.937687 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:05.937654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:05.937839 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.937810    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:08:05.937839 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.937829    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:08:05.937839 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.937838    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ww4nl for pod openshift-network-diagnostics/network-check-target-dnqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:05.937927 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:05.937887    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl podName:09bd042b-b415-46cc-a7af-aa4a57ed11d5 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:06.93787371 +0000 UTC m=+3.213309784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww4nl" (UniqueName: "kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl") pod "network-check-target-dnqj9" (UID: "09bd042b-b415-46cc-a7af-aa4a57ed11d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:06.110361 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.110282    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:06.171086 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.171021    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 13:03:05 +0000 UTC" deadline="2028-06-04 18:43:47.617036816 +0000 UTC"
Nov 03 13:08:06.171086 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.171081    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13901h35m41.445959962s"
Nov 03 13:08:06.261337 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.261238    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-72hxg" event={"ID":"3acab04e-53c1-4ceb-b571-246d23c55934","Type":"ContainerStarted","Data":"defda70f1e3cf052c5a629d93ad869aacee0508bc4d8db8ba51bd76c437f92f8"}
Nov 03 13:08:06.263099 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.263072    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k445k" event={"ID":"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642","Type":"ContainerStarted","Data":"1eb9f33307b58153cf0ce3ab50bac2a1e7e5004fc33cb2a6d87f064107d73d2e"}
Nov 03 13:08:06.265547 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.265500    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zkrzz" event={"ID":"8db0e985-d36d-485d-91ae-917f751a61a1","Type":"ContainerStarted","Data":"d3028bcd6fdccb591f9779c911708f5c164c1563ceb1607361d7731bd3840f05"}
Nov 03 13:08:06.267093 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.267069    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lx28x" event={"ID":"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01","Type":"ContainerStarted","Data":"0a61ef30690bfe69177dbda8ce018d0180dca73085a4be9b5826c651d34a2ae4"}
Nov 03 13:08:06.268742 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.268483    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t" event={"ID":"dca216a1-acaf-4fe6-b833-4fc7e25ad427","Type":"ContainerStarted","Data":"3ad193723cf185eca2af116534b69a27edc90f9ec878f7f94aa0d133122c5847"}
Nov 03 13:08:06.293728 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.293600    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" event={"ID":"a22ec490-f623-4cda-a693-e6669a1a3324","Type":"ContainerStarted","Data":"279e8c90c9de5e2c3911a2aac1346cae7ffe731d3e564ccaf0cd915f068cf39a"}
Nov 03 13:08:06.302162 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.301952    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4bn8t" event={"ID":"aa3f9f6e-77c4-4053-9289-65469d56d6ac","Type":"ContainerStarted","Data":"51c9df65e99c8e774fe7562ff97c0be38c8d184bc9565eee4dad5b27368d9c61"}
Nov 03 13:08:06.304945 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.304869    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-729bm" event={"ID":"22aec86e-8e32-4e9b-82d7-218a2ceb3926","Type":"ContainerStarted","Data":"5b01e044c6f2cf78b6d00062244c51da4915a3e081f4ac5cd50863f59cb9b8c8"}
Nov 03 13:08:06.308065 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.307998    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"1bebcb79842815c8633e47e5468f5a5a","Type":"ContainerStarted","Data":"5dc2b06e68791904bccd9bd8df8e3adf716a720cd98d047704d55b78877e53e3"}
Nov 03 13:08:06.313104 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.313076    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"be41ec8017368ca412baa7a38539e26d","Type":"ContainerStarted","Data":"24900b96e5aa02946a235af3656ad6fe62d6cad2008ca8569568f6034cd5d130"}
Nov 03 13:08:06.321150 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.321115    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xptcl" event={"ID":"c6ccec32-8bf2-4f47-8d30-abd99b513d1b","Type":"ContainerStarted","Data":"f950a5f26311d75c891f5a30bac42e0715df627a3c435b5aa890ff02f0f747c5"}
Nov 03 13:08:06.437376 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.437284    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:06.845357 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.845279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:06.845357 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.845341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:06.845579 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:06.845429    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:06.845579 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:06.845465    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:06.845579 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:06.845506    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret podName:2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df nodeName:}" failed. No retries permitted until 2026-11-03 13:08:08.845486369 +0000 UTC m=+5.120922446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret") pod "global-pull-secret-syncer-lr88l" (UID: "2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:06.845579 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:06.845530    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs podName:0d2aa382-9410-48f2-b139-3d9d12a26b3b nodeName:}" failed. No retries permitted until 2026-11-03 13:08:08.845516729 +0000 UTC m=+5.120952812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs") pod "network-metrics-daemon-5gj6w" (UID: "0d2aa382-9410-48f2-b139-3d9d12a26b3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:06.945905 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:06.945866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:06.946070 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:06.946059    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:08:06.946125 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:06.946077    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:08:06.946125 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:06.946090    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ww4nl for pod openshift-network-diagnostics/network-check-target-dnqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:06.946314 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:06.946182    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl podName:09bd042b-b415-46cc-a7af-aa4a57ed11d5 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:08.946162618 +0000 UTC m=+5.221598713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww4nl" (UniqueName: "kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl") pod "network-check-target-dnqj9" (UID: "09bd042b-b415-46cc-a7af-aa4a57ed11d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:07.005366 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:07.005337    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:08:07.171538 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:07.171437    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 13:03:05 +0000 UTC" deadline="2028-06-08 21:22:43.471428271 +0000 UTC"
Nov 03 13:08:07.171538 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:07.171491    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14000h14m36.299941395s"
Nov 03 13:08:07.254847 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:07.254366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:07.254847 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:07.254392    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:07.254847 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:07.254496    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:07.254847 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:07.254557    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:07.254847 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:07.254650    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:07.254847 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:07.254707    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:08.862757 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:08.862722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:08.863352 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:08.862789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:08.863352 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:08.862899    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:08.863352 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:08.862926    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:08.863352 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:08.862984    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret podName:2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df nodeName:}" failed. No retries permitted until 2026-11-03 13:08:12.862963498 +0000 UTC m=+9.138399574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret") pod "global-pull-secret-syncer-lr88l" (UID: "2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:08.863352 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:08.863005    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs podName:0d2aa382-9410-48f2-b139-3d9d12a26b3b nodeName:}" failed. No retries permitted until 2026-11-03 13:08:12.862993943 +0000 UTC m=+9.138430019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs") pod "network-metrics-daemon-5gj6w" (UID: "0d2aa382-9410-48f2-b139-3d9d12a26b3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:08.963812 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:08.963777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:08.964031 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:08.964013    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:08:08.964092 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:08.964037    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:08:08.964092 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:08.964050    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ww4nl for pod openshift-network-diagnostics/network-check-target-dnqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:08.964149 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:08.964108    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl podName:09bd042b-b415-46cc-a7af-aa4a57ed11d5 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:12.96409026 +0000 UTC m=+9.239526339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww4nl" (UniqueName: "kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl") pod "network-check-target-dnqj9" (UID: "09bd042b-b415-46cc-a7af-aa4a57ed11d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:09.254167 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:09.254134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:09.254340 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:09.254271    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:09.254612 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:09.254579    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:09.254720 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:09.254652    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:09.254720 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:09.254690    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:09.254849 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:09.254796    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:10.340365 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:10.339961    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"1bebcb79842815c8633e47e5468f5a5a","Type":"ContainerStarted","Data":"5ba0985e3f25fc441569710e76078af955eaacd76e8b65fbc159f7733ebd259c"}
Nov 03 13:08:10.347891 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:10.347184    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="1071962ef8ae0c6c7074bda3434258583b1ff624ce9e957c57ba500aaf41b332" exitCode=0
Nov 03 13:08:10.347891 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:10.347249    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k445k" event={"ID":"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642","Type":"ContainerDied","Data":"1071962ef8ae0c6c7074bda3434258583b1ff624ce9e957c57ba500aaf41b332"}
Nov 03 13:08:10.383929 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:10.383856    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal" podStartSLOduration=5.383837764 podStartE2EDuration="5.383837764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:10.356137126 +0000 UTC m=+6.631573222" watchObservedRunningTime="2026-11-03 13:08:10.383837764 +0000 UTC m=+6.659273861"
Nov 03 13:08:11.254729 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:11.254694    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:11.254901 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:11.254792    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:11.255095 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:11.255081    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:11.255176 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:11.255163    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:11.255256 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:11.255203    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:11.255306 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:11.255287    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:12.359771 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.359688    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4bn8t" event={"ID":"aa3f9f6e-77c4-4053-9289-65469d56d6ac","Type":"ContainerStarted","Data":"07937cc9f49ed441f408d67a0b68238b645f3d1ec5151e3bb47208c7976a69e9"}
Nov 03 13:08:12.362030 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.361997    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-729bm" event={"ID":"22aec86e-8e32-4e9b-82d7-218a2ceb3926","Type":"ContainerStarted","Data":"d6bf1dafa131465a24a7221ae39de96445ac4eac602883b66f5091606803e83a"}
Nov 03 13:08:12.365338 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.365311    2576 generic.go:356] "Generic (PLEG): container finished" podID="be41ec8017368ca412baa7a38539e26d" containerID="804f151da7bc7c982099c6b8a5c5ea470afea576edb7e9440be29c2fbad5761f" exitCode=0
Nov 03 13:08:12.365430 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.365387    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"be41ec8017368ca412baa7a38539e26d","Type":"ContainerDied","Data":"804f151da7bc7c982099c6b8a5c5ea470afea576edb7e9440be29c2fbad5761f"}
Nov 03 13:08:12.368281 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.368261    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xptcl" event={"ID":"c6ccec32-8bf2-4f47-8d30-abd99b513d1b","Type":"ContainerStarted","Data":"205aa0daf21f18e5d09028f00f8519efdecee3db00feb4e5ac9589e54deedd23"}
Nov 03 13:08:12.394120 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.394067    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4bn8t" podStartSLOduration=2.378828579 podStartE2EDuration="8.394049375s" totalImagesPullingTime="6.015220796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:12.378172997 +0000 UTC m=+8.653609094" watchObservedRunningTime="2026-11-03 13:08:12.394049375 +0000 UTC m=+8.669485472"
Nov 03 13:08:12.394570 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.394529    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-729bm" podStartSLOduration=2.095776869 podStartE2EDuration="8.394519879s" totalImagesPullingTime="6.29874301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:12.39449108 +0000 UTC m=+8.669927178" watchObservedRunningTime="2026-11-03 13:08:12.394519879 +0000 UTC m=+8.669955975"
Nov 03 13:08:12.429737 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.429671    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xptcl" podStartSLOduration=2.158769452 podStartE2EDuration="8.429652396s" totalImagesPullingTime="6.270882944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:12.410920587 +0000 UTC m=+8.686356698" watchObservedRunningTime="2026-11-03 13:08:12.429652396 +0000 UTC m=+8.705088496"
Nov 03 13:08:12.895494 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.895404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:12.895494 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.895461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:12.895704 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:12.895549    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:12.895704 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:12.895552    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:12.895704 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:12.895604    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs podName:0d2aa382-9410-48f2-b139-3d9d12a26b3b nodeName:}" failed. No retries permitted until 2026-11-03 13:08:20.895587564 +0000 UTC m=+17.171023640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs") pod "network-metrics-daemon-5gj6w" (UID: "0d2aa382-9410-48f2-b139-3d9d12a26b3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:12.895704 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:12.895618    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret podName:2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df nodeName:}" failed. No retries permitted until 2026-11-03 13:08:20.895612554 +0000 UTC m=+17.171048628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret") pod "global-pull-secret-syncer-lr88l" (UID: "2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:12.996355 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:12.996320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:12.996512 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:12.996445    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:08:12.996512 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:12.996465    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:08:12.996512 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:12.996478    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ww4nl for pod openshift-network-diagnostics/network-check-target-dnqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:12.996668 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:12.996544    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl podName:09bd042b-b415-46cc-a7af-aa4a57ed11d5 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:20.996524686 +0000 UTC m=+17.271960777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww4nl" (UniqueName: "kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl") pod "network-check-target-dnqj9" (UID: "09bd042b-b415-46cc-a7af-aa4a57ed11d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:13.254913 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:13.254846    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:13.254913 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:13.254876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:13.255120 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:13.254969    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:13.255120 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:13.255024    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:13.255120 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:13.255088    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:13.255294 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:13.255174    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:13.373599 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:13.373558    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t" event={"ID":"dca216a1-acaf-4fe6-b833-4fc7e25ad427","Type":"ContainerStarted","Data":"d31d409fcbd1b05e82ead49b53cf91e18939e9460d058ae2a2e176d572316c67"}
Nov 03 13:08:13.376423 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:13.375825    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"be41ec8017368ca412baa7a38539e26d","Type":"ContainerStarted","Data":"bb6f5730b3c12611b316a0830442bf9add9a2bdc68d9be5ee987710388838d6f"}
Nov 03 13:08:13.391519 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:13.391481    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal" podStartSLOduration=8.39146671 podStartE2EDuration="8.39146671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:13.391005864 +0000 UTC m=+9.666441960" watchObservedRunningTime="2026-11-03 13:08:13.39146671 +0000 UTC m=+9.666902808"
Nov 03 13:08:14.379503 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:14.379420    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lx28x" event={"ID":"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01","Type":"ContainerStarted","Data":"76b80f5755668381e0226ec15e8173a64fdf29b5519d6c7345f9ea5f61e05dd4"}
Nov 03 13:08:14.382108 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:14.382082    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zkrzz" event={"ID":"8db0e985-d36d-485d-91ae-917f751a61a1","Type":"ContainerStarted","Data":"334af3d4634d5d15bef6a33fc543833c3299f7b691005536edbccecc0fc31df9"}
Nov 03 13:08:14.394513 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:14.394476    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lx28x" podStartSLOduration=2.288536829 podStartE2EDuration="10.394465378s" totalImagesPullingTime="8.105928549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:14.394112575 +0000 UTC m=+10.669548671" watchObservedRunningTime="2026-11-03 13:08:14.394465378 +0000 UTC m=+10.669901473"
Nov 03 13:08:14.960155 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:14.960074    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 13:08:15.185021 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.184851    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:08:14.960096818Z","UUID":"b667b67c-6c68-4f2d-8d8b-933dba70670a","Handler":null,"Name":"","Endpoint":""}
Nov 03 13:08:15.186739 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.186718    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 13:08:15.186868 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.186752    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 13:08:15.254621 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.254591    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:15.254781 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.254638    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:15.254781 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:15.254709    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:15.254888 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:15.254779    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:15.254888 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.254846    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:15.254984 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:15.254935    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:15.386828 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.386791    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t" event={"ID":"dca216a1-acaf-4fe6-b833-4fc7e25ad427","Type":"ContainerStarted","Data":"58609dab4b157c9ceeebe2deb668449f6c23af3d57dd1e2e9c4ca37f54d9d884"}
Nov 03 13:08:15.427106 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.426877    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:15.513582 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.513506    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:15.514231 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.514191    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:15.544800 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.544755    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zkrzz" podStartSLOduration=4.365305066 podStartE2EDuration="11.544738475s" totalImagesPullingTime="7.179433409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:14.412554177 +0000 UTC m=+10.687990284" watchObservedRunningTime="2026-11-03 13:08:15.544738475 +0000 UTC m=+11.820174573"
Nov 03 13:08:15.688958 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.688930    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:15.818477 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:15.818444    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:16.390261 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:16.390225    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-72hxg" event={"ID":"3acab04e-53c1-4ceb-b571-246d23c55934","Type":"ContainerStarted","Data":"74728f8b56039908b34b243e0ae7e7b8e140e83c9ea779c214fb7c53508a90de"}
Nov 03 13:08:16.392017 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:16.391992    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="c0de7afca2a06488f054cd538633d621fe6978db272f76e3b0a0b17648e34c65" exitCode=0
Nov 03 13:08:16.392116 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:16.392075    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k445k" event={"ID":"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642","Type":"ContainerDied","Data":"c0de7afca2a06488f054cd538633d621fe6978db272f76e3b0a0b17648e34c65"}
Nov 03 13:08:16.407541 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:16.407499    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-72hxg" podStartSLOduration=2.246257079 podStartE2EDuration="12.407487084s" totalImagesPullingTime="10.161230005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:16.406798202 +0000 UTC m=+12.682234297" watchObservedRunningTime="2026-11-03 13:08:16.407487084 +0000 UTC m=+12.682923180"
Nov 03 13:08:17.254825 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:17.254799    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:17.254974 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:17.254889    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:17.254974 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:17.254904    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:17.254974 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:17.254918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:17.255058 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:17.254991    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:17.255058 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:17.255051    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:18.397201 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:18.396967    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t" event={"ID":"dca216a1-acaf-4fe6-b833-4fc7e25ad427","Type":"ContainerStarted","Data":"1c3b9f031690274834327f486c7dda7f4eff14de306bd1f61e74eeeaeea4638d"}
Nov 03 13:08:18.399030 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:18.399008    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="be55a4e2642c3f9745e252ae26d30a978510c0fd68c207054473608456dcfbde" exitCode=0
Nov 03 13:08:18.399129 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:18.399045    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k445k" event={"ID":"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642","Type":"ContainerDied","Data":"be55a4e2642c3f9745e252ae26d30a978510c0fd68c207054473608456dcfbde"}
Nov 03 13:08:18.415573 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:18.415534    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t" podStartSLOduration=2.445583073 podStartE2EDuration="14.415521335s" totalImagesPullingTime="11.969938262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:18.413533061 +0000 UTC m=+14.688969181" watchObservedRunningTime="2026-11-03 13:08:18.415521335 +0000 UTC m=+14.690957464"
Nov 03 13:08:19.064165 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:19.064123    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:19.202037 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:19.202009    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:19.254794 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:19.254768    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:19.254917 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:19.254798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:19.254917 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:19.254866    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:19.254990 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:19.254912    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:19.254990 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:19.254931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:19.255056 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:19.255014    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:20.405430 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:20.405390    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="bcb3044779e1819ac303837ae0c471ffa896e64362a01f29533c5d5ca3e4b975" exitCode=0
Nov 03 13:08:20.405915 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:20.405441    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k445k" event={"ID":"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642","Type":"ContainerDied","Data":"bcb3044779e1819ac303837ae0c471ffa896e64362a01f29533c5d5ca3e4b975"}
Nov 03 13:08:20.958598 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:20.958554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:20.958802 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:20.958606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:20.958802 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:20.958714    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:20.958802 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:20.958773    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs podName:0d2aa382-9410-48f2-b139-3d9d12a26b3b nodeName:}" failed. No retries permitted until 2026-11-03 13:08:36.958759816 +0000 UTC m=+33.234195893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs") pod "network-metrics-daemon-5gj6w" (UID: "0d2aa382-9410-48f2-b139-3d9d12a26b3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:20.958905 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:20.958714    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:20.958905 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:20.958847    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret podName:2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df nodeName:}" failed. No retries permitted until 2026-11-03 13:08:36.958834964 +0000 UTC m=+33.234271052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret") pod "global-pull-secret-syncer-lr88l" (UID: "2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:21.059554 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:21.059518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:21.059720 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:21.059653    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:08:21.059720 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:21.059668    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:08:21.059720 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:21.059677    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ww4nl for pod openshift-network-diagnostics/network-check-target-dnqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:21.059877 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:21.059738    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl podName:09bd042b-b415-46cc-a7af-aa4a57ed11d5 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:37.059724854 +0000 UTC m=+33.335160931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww4nl" (UniqueName: "kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl") pod "network-check-target-dnqj9" (UID: "09bd042b-b415-46cc-a7af-aa4a57ed11d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:21.254911 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:21.254831    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:21.255054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:21.254831    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:21.255054 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:21.254950    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:21.255054 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:21.254831    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:21.255054 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:21.255021    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:21.255173 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:21.255080    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:22.305089 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:22.305056    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:22.819833 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:22.819805    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:22.950458 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:22.950429    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:23.254965 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:23.254886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:23.254965 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:23.254903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:23.255150 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:23.254903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:23.255150 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:23.255017    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:23.255150 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:23.255088    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:23.255264 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:23.255204    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:25.254627 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:25.254595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:25.254627 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:25.254624    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:25.255039 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:25.254689    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:25.255039 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:25.254731    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:25.255039 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:25.254822    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:25.255039 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:25.254884    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:25.365541 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:25.365503    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:25.489670 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:25.489638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:26.834572 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:26.834541    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:27.254878 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:27.254843    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:27.254878 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:27.254874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:27.255129 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:27.254874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:27.255129 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:27.254951    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:27.255129 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:27.255059    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:27.255270 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:27.255158    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:29.254188 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:29.254138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:29.254781 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:29.254233    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:29.254781 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:29.254329    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:29.254781 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:29.254352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:29.254781 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:29.254395    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:29.254781 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:29.254452    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:29.627608 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:29.627535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:29.754143 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:29.754108    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:30.586366 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:30.586340    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lx28x_1ffdebd2-3a79-4eb9-baf4-87b91d78ac01/dns-node-resolver/0.log"
Nov 03 13:08:31.255040 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:31.255006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:31.255312 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:31.255049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:31.255312 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:31.255095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:31.255312 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:31.255111    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:31.255312 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:31.255169    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:31.255312 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:31.255246    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:31.416135 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:31.416102    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:31.416332 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:31.416255    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 13:08:31.416691 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:31.416673    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:08:31.435194 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:31.435166    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:31.560298 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:31.560227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:31.786022 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:31.785994    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xptcl_c6ccec32-8bf2-4f47-8d30-abd99b513d1b/node-ca/0.log"
Nov 03 13:08:32.353973 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:32.353944    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:32.480879 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:32.480847    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:32.608047 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:32.607965    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:33.073599 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:33.073563    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:33.211617 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:33.211588    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:33.255034 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:33.255005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:33.255170 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:33.255005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:33.255170 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:33.255120    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:33.255260 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:33.255005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:33.255260 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:33.255186    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:33.255321 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:33.255274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:33.584730 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:33.584701    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:35.254827 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:35.254794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:35.255194 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:35.254795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:35.255194 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:35.254894    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:35.255194 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:35.254920    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:35.255194 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:35.254993    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:35.255194 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:35.255070    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:35.435662 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:35.435632    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="0ff963af78644ae46cffe1209cc5993ab2c4890093e0d2d7569976c702985cb6" exitCode=0
Nov 03 13:08:35.435816 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:35.435669    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k445k" event={"ID":"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642","Type":"ContainerDied","Data":"0ff963af78644ae46cffe1209cc5993ab2c4890093e0d2d7569976c702985cb6"}
Nov 03 13:08:36.247268 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:36.247236    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:36.440385 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:36.440354    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="446ef6194a788dc976dd1d623bec2f8ce8bf712cafd38b7587a6665f18b2dcc9" exitCode=0
Nov 03 13:08:36.440801 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:36.440406    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k445k" event={"ID":"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642","Type":"ContainerDied","Data":"446ef6194a788dc976dd1d623bec2f8ce8bf712cafd38b7587a6665f18b2dcc9"}
Nov 03 13:08:36.907582 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:36.907505    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:36.965907 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:36.965880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:36.966057 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:36.965915    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:36.966057 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:36.966029    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:36.966124 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:36.966085    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs podName:0d2aa382-9410-48f2-b139-3d9d12a26b3b nodeName:}" failed. No retries permitted until 2026-11-03 13:09:08.966071535 +0000 UTC m=+65.241507609 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs") pod "network-metrics-daemon-5gj6w" (UID: "0d2aa382-9410-48f2-b139-3d9d12a26b3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 13:08:36.966124 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:36.966029    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:36.966190 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:36.966158    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret podName:2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df nodeName:}" failed. No retries permitted until 2026-11-03 13:09:08.966146212 +0000 UTC m=+65.241582300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret") pod "global-pull-secret-syncer-lr88l" (UID: "2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 13:08:37.043336 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.043306    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:37.066959 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.066928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:37.067079 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:37.067040    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 13:08:37.067079 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:37.067052    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 13:08:37.067079 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:37.067061    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ww4nl for pod openshift-network-diagnostics/network-check-target-dnqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:37.067173 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:37.067105    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl podName:09bd042b-b415-46cc-a7af-aa4a57ed11d5 nodeName:}" failed. No retries permitted until 2026-11-03 13:09:09.067094179 +0000 UTC m=+65.342530266 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ww4nl" (UniqueName: "kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl") pod "network-check-target-dnqj9" (UID: "09bd042b-b415-46cc-a7af-aa4a57ed11d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 13:08:37.254201 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.254169    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:37.254386 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.254245    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:37.254386 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.254326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:37.254386 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:37.254336    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:37.254532 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:37.254387    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:37.254532 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:37.254484    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:37.313567 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.313532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:37.445599 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.445570    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k445k" event={"ID":"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642","Type":"ContainerStarted","Data":"12bdef35019d667db590c95a9e2dffb190de3bff39070d55eaaf74f1353ab9a8"}
Nov 03 13:08:37.465731 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.465686    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k445k" podStartSLOduration=8.507758166 podStartE2EDuration="33.465673523s" totalImagesPullingTime="24.957915357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:37.465328159 +0000 UTC m=+33.740764256" watchObservedRunningTime="2026-11-03 13:08:37.465673523 +0000 UTC m=+33.741109619"
Nov 03 13:08:37.504388 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.504317    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:37.904924 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:37.904851    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:38.701823 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:38.701793    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:39.107628 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:39.107593    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:39.254005 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:39.253975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:39.254163 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:39.253982    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:39.254163 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:39.254084    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:39.254163 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:39.254078    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:39.254163 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:39.254154    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:39.254315 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:39.254249    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:39.311413 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:39.311384    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:39.499552 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:39.499520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:39.904005 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:39.903926    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:40.063513 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:40.063484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:40.705537 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:40.705474    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:41.163083 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:41.163005    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:41.254223 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:41.254186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:41.254395 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:41.254246    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:41.254395 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:41.254195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:41.254395 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:41.254316    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:41.254395 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:41.254375    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:41.254533 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:41.254444    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:41.298726 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:41.298694    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:41.511900 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:41.511870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:41.911119 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:41.911047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:42.061711 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:42.061678    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:42.199288 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:42.199200    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:42.321947 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:42.321917    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:42.714428 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:42.714398    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:42.855744 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:42.855711    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:43.010990 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.010919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:43.139603 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.139575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:43.254227 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.254187    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:43.254227 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.254204    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:43.254637 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.254196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:43.254637 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:43.254313    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:43.254637 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:43.254368    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:43.254637 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:43.254468    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:43.262734 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.262676    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:43.387955 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.387925    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:43.520816 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.520747    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:43.841339 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.841262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:43.968444 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:43.968410    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:44.092205 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:44.092132    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:44.641049 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:44.641011    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:44.782665 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:44.782630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:44.917358 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:44.917284    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:45.075504 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:45.075467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:45.211599 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:45.211569    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:45.254306 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:45.254274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:45.254306 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:45.254309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:45.254540 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:45.254280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:45.254540 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:45.254400    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:45.254540 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:45.254526    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:45.254689 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:45.254592    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:45.363672 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:45.363638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:45.589425 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:45.589348    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:45.946121 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:45.946037    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:46.723677 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:46.723644    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:47.133233 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:47.133125    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:47.254121 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:47.254082    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:47.254326 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:47.254087    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:47.254326 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:47.254196    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:47.254326 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:47.254309    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:47.254449 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:47.254087    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:47.254449 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:47.254423    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:47.305789 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:47.305762    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:47.501251 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:47.501206    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:47.902110 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:47.902038    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:48.109680 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:48.109645    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:49.254367 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:49.254331    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:49.254810 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:49.254442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:49.254810 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:49.254551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:49.254810 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:49.254567    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:49.254810 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:49.254658    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:49.254810 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:49.254725    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:51.254301 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:51.254269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:51.254742 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:51.254374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:51.254742 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:51.254386    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:51.254742 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:51.254516    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:51.254742 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:51.254547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:51.254742 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:51.254615    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:52.479739 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:52.479530    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" event={"ID":"a22ec490-f623-4cda-a693-e6669a1a3324","Type":"ContainerStarted","Data":"143e15ea42e8890d6dbae173327334583597438cb9a33b7ac63729ccdd15d1b4"}
Nov 03 13:08:52.480488 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:52.479829    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" event={"ID":"a22ec490-f623-4cda-a693-e6669a1a3324","Type":"ContainerStarted","Data":"c67cef2afc6caeec2a319c5799f737eb99537478c6f3150cf71bbbab8e09591d"}
Nov 03 13:08:52.480488 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:52.479875    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" event={"ID":"a22ec490-f623-4cda-a693-e6669a1a3324","Type":"ContainerStarted","Data":"9ed8f9743bbb3678001349b25c70af93c0c041229a6e9d233bd011b7c3edcaf6"}
Nov 03 13:08:52.480488 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:52.479915    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" event={"ID":"a22ec490-f623-4cda-a693-e6669a1a3324","Type":"ContainerStarted","Data":"caae49e598fd497a444111d52bf1042460f5e816ec426cd728035c4fc665fdd6"}
Nov 03 13:08:52.480488 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:52.479953    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" event={"ID":"a22ec490-f623-4cda-a693-e6669a1a3324","Type":"ContainerStarted","Data":"3aeb7199c3e9a9047ea978fee4f16645578e7dd08377d721ecfde90f4aac173d"}
Nov 03 13:08:52.480488 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:52.480002    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" event={"ID":"a22ec490-f623-4cda-a693-e6669a1a3324","Type":"ContainerStarted","Data":"e3cb03f0be3381acfa5266594ccd241a0458951623155597f434065c7db85bb0"}
Nov 03 13:08:53.254173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:53.254133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:53.254173 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:53.254175    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:53.254382 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:53.254141    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:53.254382 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:53.254268    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:53.254382 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:53.254327    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:53.254475 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:53.254388    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:53.622176 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:53.622093    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:53.796192 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:53.796164    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:55.254491 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:55.254464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:55.254873 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:55.254464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:55.254873 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:55.254565    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:55.254873 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:55.254464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:55.254873 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:55.254638    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:55.254873 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:55.254733    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:55.489411 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:55.489376    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" event={"ID":"a22ec490-f623-4cda-a693-e6669a1a3324","Type":"ContainerStarted","Data":"e28cb986433ff5d8d3d9be22553a562336ad5f9dbedbb762217c188ff5e6bcdf"}
Nov 03 13:08:55.873784 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:55.873752    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lx28x_1ffdebd2-3a79-4eb9-baf4-87b91d78ac01/dns-node-resolver/0.log"
Nov 03 13:08:56.118416 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:56.118382    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:56.268145 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:56.268119    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:56.445390 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:56.445358    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:57.254820 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.254638    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:57.254976 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.254708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:57.254976 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:57.254902    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:57.254976 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.254713    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:57.255086 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:57.254976    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:57.255086 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:57.255035    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:57.274410 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.274390    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xptcl_c6ccec32-8bf2-4f47-8d30-abd99b513d1b/node-ca/0.log"
Nov 03 13:08:57.497163 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.497131    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" event={"ID":"a22ec490-f623-4cda-a693-e6669a1a3324","Type":"ContainerStarted","Data":"f6a6172096a163401ee35c15068c507c0f7bc8612edf254851f3ea7ba87aa18d"}
Nov 03 13:08:57.497442 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.497422    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:57.497495 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.497481    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:57.498691 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.498667    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:08:57.498796 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.498695    2576 generic.go:356] "Generic (PLEG): container finished" podID="aa3f9f6e-77c4-4053-9289-65469d56d6ac" containerID="07937cc9f49ed441f408d67a0b68238b645f3d1ec5151e3bb47208c7976a69e9" exitCode=1
Nov 03 13:08:57.498796 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.498734    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4bn8t" event={"ID":"aa3f9f6e-77c4-4053-9289-65469d56d6ac","Type":"ContainerDied","Data":"07937cc9f49ed441f408d67a0b68238b645f3d1ec5151e3bb47208c7976a69e9"}
Nov 03 13:08:57.498986 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.498974    2576 scope.go:122] "RemoveContainer" containerID="07937cc9f49ed441f408d67a0b68238b645f3d1ec5151e3bb47208c7976a69e9"
Nov 03 13:08:57.513751 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.513732    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:57.530532 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:57.530495    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp" podStartSLOduration=6.959901273 podStartE2EDuration="53.530467692s" totalImagesPullingTime="46.570566419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:57.528762017 +0000 UTC m=+53.804198114" watchObservedRunningTime="2026-11-03 13:08:57.530467692 +0000 UTC m=+53.805903778"
Nov 03 13:08:58.243745 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.243059    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dnqj9"]
Nov 03 13:08:58.243745 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.243450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:08:58.243745 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:58.243543    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:08:58.244010 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.243865    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lr88l"]
Nov 03 13:08:58.244010 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.243989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:08:58.244100 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:58.244078    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:08:58.246253 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.246184    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5gj6w"]
Nov 03 13:08:58.246446 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.246428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:58.246687 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:58.246664    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:08:58.429679 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.429591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:58.502464 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.502436    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:08:58.502642 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.502501    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4bn8t" event={"ID":"aa3f9f6e-77c4-4053-9289-65469d56d6ac","Type":"ContainerStarted","Data":"fbac89ec8ecd77973670c530b839c6cce80afad9d998ef0c42ed9bbd7a571cf5"}
Nov 03 13:08:58.502918 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.502904    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:58.518423 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:58.518398    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:08:59.254673 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:08:59.254632    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:08:59.254943 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:08:59.254765    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:09:00.254336 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.254303    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:00.254740 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:09:00.254439    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:09:00.254740 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.254476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:00.254740 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:09:00.254561    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:09:00.272281 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.272257    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:00.489266 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.489234    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ggskq"]
Nov 03 13:09:00.492540 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.492506    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.508407 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.508345    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 13:09:00.509975 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.509951    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 13:09:00.509975 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.509968    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjbx2\""
Nov 03 13:09:00.514940 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.514923    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:00.589593 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.589560    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 13:09:00.589593 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.589578    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 13:09:00.589789 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.589627    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 13:09:00.589789 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.589654    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 13:09:00.627532 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.627494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-textfile\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.627705 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.627544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.627705 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.627682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-root\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.627782 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.627725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.627850 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.627829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-tls\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.628120 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.628098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-sys\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.628205 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.628152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-wtmp\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.628294 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.628279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-metrics-client-ca\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.628403 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.628385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6rh\" (UniqueName: \"kubernetes.io/projected/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-kube-api-access-st6rh\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.662969 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.662941    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:00.729062 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-root\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729243 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729243 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-tls\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729243 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-sys\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729243 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-wtmp\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729243 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729174    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-root\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729243 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-metrics-client-ca\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729243 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729242    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6rh\" (UniqueName: \"kubernetes.io/projected/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-kube-api-access-st6rh\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729585 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729275    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-sys\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729585 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-textfile\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729585 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729585 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729347    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-wtmp\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729739 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729629    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-textfile\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729984 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729955    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.729984 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.729971    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-metrics-client-ca\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.734100 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.734078    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.734100 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.734089    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-tls\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.737694 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.737672    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6rh\" (UniqueName: \"kubernetes.io/projected/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-kube-api-access-st6rh\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.803113 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.803049    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:00.811570 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.811547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:00.822722 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:09:00.822689    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f20bcca_4ad5_4241_bf25_c59211b8eb9c.slice/crio-9b65dadc004a34cdc6bcc15e20663bc5921b5e4bcd3b5cd163e09a289dc9c941 WatchSource:0}: Error finding container 9b65dadc004a34cdc6bcc15e20663bc5921b5e4bcd3b5cd163e09a289dc9c941: Status 404 returned error can't find the container with id 9b65dadc004a34cdc6bcc15e20663bc5921b5e4bcd3b5cd163e09a289dc9c941
Nov 03 13:09:00.955369 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:00.955338    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:01.084092 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:01.084014    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:01.254275 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:01.254246    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:01.254409 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:09:01.254359    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:09:01.511094 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:01.511056    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ggskq" event={"ID":"0f20bcca-4ad5-4241-bf25-c59211b8eb9c","Type":"ContainerStarted","Data":"9b65dadc004a34cdc6bcc15e20663bc5921b5e4bcd3b5cd163e09a289dc9c941"}
Nov 03 13:09:01.522991 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:01.522964    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:01.663899 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:01.663817    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:02.254807 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:02.254604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:02.255237 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:02.254639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:02.255237 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:09:02.254897    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:09:02.255237 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:09:02.254947    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:09:02.515431 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:02.515330    2576 generic.go:356] "Generic (PLEG): container finished" podID="0f20bcca-4ad5-4241-bf25-c59211b8eb9c" containerID="694028e85bd36a494c34429b23be4f9647d646b2e747f5335251007ed1efe4dd" exitCode=0
Nov 03 13:09:02.515431 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:02.515415    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ggskq" event={"ID":"0f20bcca-4ad5-4241-bf25-c59211b8eb9c","Type":"ContainerDied","Data":"694028e85bd36a494c34429b23be4f9647d646b2e747f5335251007ed1efe4dd"}
Nov 03 13:09:02.665950 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:02.665914    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:02.800008 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:02.799921    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:02.966826 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:02.966789    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:03.100840 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:03.100759    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:03.254851 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:03.254828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:03.255203 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:09:03.254932    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gj6w" podUID="0d2aa382-9410-48f2-b139-3d9d12a26b3b"
Nov 03 13:09:03.520199 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:03.520163    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ggskq" event={"ID":"0f20bcca-4ad5-4241-bf25-c59211b8eb9c","Type":"ContainerStarted","Data":"bd89cfe74b85e9802e0e9af471ef8b28a7e09cb3c546308d8d806bff4197a65c"}
Nov 03 13:09:03.520378 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:03.520236    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ggskq" event={"ID":"0f20bcca-4ad5-4241-bf25-c59211b8eb9c","Type":"ContainerStarted","Data":"43ce1574c9fe66b980eeefd4ec5829bfeffdcc7bd9caa155cb651bf8328e2296"}
Nov 03 13:09:03.549578 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:03.549550    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:03.743397 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:03.743365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:03.952100 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:03.952024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:04.229671 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:04.229644    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:04.255486 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:04.255452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:04.255989 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:09:04.255546    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dnqj9" podUID="09bd042b-b415-46cc-a7af-aa4a57ed11d5"
Nov 03 13:09:04.255989 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:04.255584    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:04.255989 ip-10-0-139-248 kubenswrapper[2576]: E1103 13:09:04.255670    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lr88l" podUID="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df"
Nov 03 13:09:04.365539 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:04.365504    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:05.065092 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.065062    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeReady"
Nov 03 13:09:05.065274 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.065184    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 13:09:05.112245 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.112179    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ggskq" podStartSLOduration=4.12615989 podStartE2EDuration="5.11216606s" totalImagesPullingTime="918.726353ms" totalInitContainerRuntime="67.279817ms" isStatefulPod=true podCreationTimestamp="2026-11-03 13:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:03.543329334 +0000 UTC m=+59.818765430" watchObservedRunningTime="2026-11-03 13:09:05.11216606 +0000 UTC m=+61.387602155"
Nov 03 13:09:05.112714 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.112702    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qdgff"]
Nov 03 13:09:05.116758 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.116738    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zfk5q"]
Nov 03 13:09:05.116897 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.116871    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.120350 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.120332    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:05.123630 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.123588    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 13:09:05.123630 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.123589    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wd8kt\""
Nov 03 13:09:05.124779 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.124758    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdgff"]
Nov 03 13:09:05.127810 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.127787    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zfk5q"]
Nov 03 13:09:05.163911 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.163888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1705c77-7f46-4719-8dd1-51993028ab1e-tmp-dir\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.163911 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.163916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1705c77-7f46-4719-8dd1-51993028ab1e-config-volume\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.164101 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.163932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2mh\" (UniqueName: \"kubernetes.io/projected/e1705c77-7f46-4719-8dd1-51993028ab1e-kube-api-access-jh2mh\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.164101 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.163994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c40982af-b5a4-4f58-b35e-93208bfd793e-cert\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:05.164101 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.164046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4ml\" (UniqueName: \"kubernetes.io/projected/c40982af-b5a4-4f58-b35e-93208bfd793e-kube-api-access-dw4ml\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:05.164232 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.164100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1705c77-7f46-4719-8dd1-51993028ab1e-metrics-tls\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.215858 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.215823    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gkggt\""
Nov 03 13:09:05.215858 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.215858    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 13:09:05.216049 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.215931    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 13:09:05.221039 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.221019    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l8wlg"]
Nov 03 13:09:05.223251 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.223237    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 13:09:05.223322 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.223304    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 13:09:05.224566 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.224553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.234579 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.234554    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s2gms\""
Nov 03 13:09:05.234668 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.234583    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 13:09:05.235335 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.235315    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8wlg"]
Nov 03 13:09:05.254755 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.254731    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:05.259328 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.259312    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-569wf\""
Nov 03 13:09:05.259613 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.259364    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 13:09:05.264555 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1705c77-7f46-4719-8dd1-51993028ab1e-config-volume\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.264655 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2mh\" (UniqueName: \"kubernetes.io/projected/e1705c77-7f46-4719-8dd1-51993028ab1e-kube-api-access-jh2mh\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.264655 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67dddf6c-5f0c-4894-aeb2-c26135c32c07-data-volume\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.264655 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c40982af-b5a4-4f58-b35e-93208bfd793e-cert\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:05.264872 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67dddf6c-5f0c-4894-aeb2-c26135c32c07-crio-socket\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.264872 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4ml\" (UniqueName: \"kubernetes.io/projected/c40982af-b5a4-4f58-b35e-93208bfd793e-kube-api-access-dw4ml\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:05.264872 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1705c77-7f46-4719-8dd1-51993028ab1e-metrics-tls\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.264993 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6x6m\" (UniqueName: \"kubernetes.io/projected/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-api-access-m6x6m\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.264993 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.264993 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264962    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1705c77-7f46-4719-8dd1-51993028ab1e-tmp-dir\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.264993 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.264986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67dddf6c-5f0c-4894-aeb2-c26135c32c07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.265128 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.265042    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1705c77-7f46-4719-8dd1-51993028ab1e-config-volume\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.265269 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.265251    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1705c77-7f46-4719-8dd1-51993028ab1e-tmp-dir\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.267914 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.267893    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1705c77-7f46-4719-8dd1-51993028ab1e-metrics-tls\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.268061 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.268044    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c40982af-b5a4-4f58-b35e-93208bfd793e-cert\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:05.273676 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.273651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4ml\" (UniqueName: \"kubernetes.io/projected/c40982af-b5a4-4f58-b35e-93208bfd793e-kube-api-access-dw4ml\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:05.273676 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.273673    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2mh\" (UniqueName: \"kubernetes.io/projected/e1705c77-7f46-4719-8dd1-51993028ab1e-kube-api-access-jh2mh\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.278795 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.278778    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"]
Nov 03 13:09:05.281838 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.281823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:05.291193 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.291169    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"]
Nov 03 13:09:05.322249 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.322145    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:05.323262 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.323244    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 13:09:05.323368 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.323326    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 13:09:05.323368 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.323353    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 13:09:05.360517 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.360489    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 13:09:05.360617 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.360605    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mfl7n\""
Nov 03 13:09:05.365958 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.365942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6x6m\" (UniqueName: \"kubernetes.io/projected/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-api-access-m6x6m\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.366019 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.365967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.366019 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.365992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67dddf6c-5f0c-4894-aeb2-c26135c32c07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.366088 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.366020    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67dddf6c-5f0c-4894-aeb2-c26135c32c07-data-volume\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.366120 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.366083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67dddf6c-5f0c-4894-aeb2-c26135c32c07-crio-socket\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.366120 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.366111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7zln\" (UniqueName: \"kubernetes.io/projected/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-kube-api-access-x7zln\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:05.366202 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.366148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:05.366359 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.366341    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67dddf6c-5f0c-4894-aeb2-c26135c32c07-crio-socket\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.366520 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.366504    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67dddf6c-5f0c-4894-aeb2-c26135c32c07-data-volume\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.366622 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.366606    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.368365 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.368348    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67dddf6c-5f0c-4894-aeb2-c26135c32c07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.374139 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.374121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6x6m\" (UniqueName: \"kubernetes.io/projected/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-api-access-m6x6m\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.433068 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.433048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:05.435922 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.435905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:05.467573 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.467539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zln\" (UniqueName: \"kubernetes.io/projected/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-kube-api-access-x7zln\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:05.467749 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.467594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:05.470034 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.470009    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:05.479885 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.479858    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zln\" (UniqueName: \"kubernetes.io/projected/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-kube-api-access-x7zln\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:05.481820 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.481786    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:05.541102 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.541078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:05.592493 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.592468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:05.618645 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.618619    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:05.726618 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.726526    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdgff"]
Nov 03 13:09:05.729462 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.729435    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zfk5q"]
Nov 03 13:09:05.730417 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:09:05.730392    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1705c77_7f46_4719_8dd1_51993028ab1e.slice/crio-3f46bdfe18ad7bd52c491497087779518ca247162bfeb9adca48933c258e29c5 WatchSource:0}: Error finding container 3f46bdfe18ad7bd52c491497087779518ca247162bfeb9adca48933c258e29c5: Status 404 returned error can't find the container with id 3f46bdfe18ad7bd52c491497087779518ca247162bfeb9adca48933c258e29c5
Nov 03 13:09:05.732628 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:09:05.732603    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40982af_b5a4_4f58_b35e_93208bfd793e.slice/crio-dd372399011085a16130308c4f082f633364c6622744300442628c4bf557c220 WatchSource:0}: Error finding container dd372399011085a16130308c4f082f633364c6622744300442628c4bf557c220: Status 404 returned error can't find the container with id dd372399011085a16130308c4f082f633364c6622744300442628c4bf557c220
Nov 03 13:09:05.809361 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.809332    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8wlg"]
Nov 03 13:09:05.812141 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:09:05.812118    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dddf6c_5f0c_4894_aeb2_c26135c32c07.slice/crio-a90b2cfd4cb781a986a7e3cb4b975dc644532151aa3dd9c8fb3202fd8a5f0b49 WatchSource:0}: Error finding container a90b2cfd4cb781a986a7e3cb4b975dc644532151aa3dd9c8fb3202fd8a5f0b49: Status 404 returned error can't find the container with id a90b2cfd4cb781a986a7e3cb4b975dc644532151aa3dd9c8fb3202fd8a5f0b49
Nov 03 13:09:05.861412 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:05.861391    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"]
Nov 03 13:09:05.864817 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:09:05.864797    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3d0acf_5ab2_47ce_99ef_21f5565fbd99.slice/crio-ebbe8aca8402f1038ae4c63f32498a1bb680533d8d15134bf6160b68ab7f08b4 WatchSource:0}: Error finding container ebbe8aca8402f1038ae4c63f32498a1bb680533d8d15134bf6160b68ab7f08b4: Status 404 returned error can't find the container with id ebbe8aca8402f1038ae4c63f32498a1bb680533d8d15134bf6160b68ab7f08b4
Nov 03 13:09:06.254638 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.254610    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:06.254821 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.254788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:06.262109 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.262083    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 13:09:06.262452 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.262108    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hltl5\""
Nov 03 13:09:06.263853 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.263834    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 13:09:06.263853 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.263842    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d95tp\""
Nov 03 13:09:06.264006 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.263906    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 13:09:06.456517 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.456333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:06.532617 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.532532    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8wlg" event={"ID":"67dddf6c-5f0c-4894-aeb2-c26135c32c07","Type":"ContainerStarted","Data":"f8c1b15f2b2db19356ea8ba3e5b1cab78592942d70117cce7ce9a12a7e661f81"}
Nov 03 13:09:06.532617 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.532605    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8wlg" event={"ID":"67dddf6c-5f0c-4894-aeb2-c26135c32c07","Type":"ContainerStarted","Data":"a90b2cfd4cb781a986a7e3cb4b975dc644532151aa3dd9c8fb3202fd8a5f0b49"}
Nov 03 13:09:06.534021 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.533989    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdgff" event={"ID":"e1705c77-7f46-4719-8dd1-51993028ab1e","Type":"ContainerStarted","Data":"3f46bdfe18ad7bd52c491497087779518ca247162bfeb9adca48933c258e29c5"}
Nov 03 13:09:06.535950 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.535404    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz" event={"ID":"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99","Type":"ContainerStarted","Data":"ebbe8aca8402f1038ae4c63f32498a1bb680533d8d15134bf6160b68ab7f08b4"}
Nov 03 13:09:06.536984 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.536960    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zfk5q" event={"ID":"c40982af-b5a4-4f58-b35e-93208bfd793e","Type":"ContainerStarted","Data":"dd372399011085a16130308c4f082f633364c6622744300442628c4bf557c220"}
Nov 03 13:09:06.657129 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.657096    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:06.791930 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.791842    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 13:09:06.795894 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.795869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.806026 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.805804    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 13:09:06.806026 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.805952    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 13:09:06.806424 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.806059    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 13:09:06.806424 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.806134    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 13:09:06.806424 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.806184    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 13:09:06.811402 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.811376    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 13:09:06.811516 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.811453    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 13:09:06.811516 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.811498    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 13:09:06.811851 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.811801    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 13:09:06.811953 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.811934    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-uhmup5ug6m5s\""
Nov 03 13:09:06.818651 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.818439    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 13:09:06.818966 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.818787    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 13:09:06.818966 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.818874    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 13:09:06.818966 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.818882    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 13:09:06.818966 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.818945    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 13:09:06.856690 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.856658    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:06.880074 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880248 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880248 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880351 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880270    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880351 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880302    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880420 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880457 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880492 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880521 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880563 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880597 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880635 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880670 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880699 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880699 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880690    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5cj\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880757 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880757 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880815 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880849 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.880884 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.880866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.906230 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.906183    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 13:09:06.906395 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.906275    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 13:09:06.906470 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.906433    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-85dfg\""
Nov 03 13:09:06.981412 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981412 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5cj\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981632 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981632 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981632 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981632 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981632 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981632 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981632 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981958 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981958 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981958 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981958 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981958 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981958 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.981958 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.981846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.982475 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.982260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.982697 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.982663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.982847 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.982828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.983055 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.983031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.993324 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.993287    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.996620 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.996593    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.997504 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.997484    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.998727 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.998704    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.998794 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.998770    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:06.998825 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:06.998801    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.000116 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.000088    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.000669 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.000643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.001154 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.001125    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:07.005583 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.005556    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.006593 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.006569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.010626 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.009308    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.010626 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.009701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.010626 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.010059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.011112 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.011071    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.013733 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.013683    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.015232 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.014495    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.015367 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.015340    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.015471 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.015387    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.017091 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.017042    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.017997 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.017956    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5cj\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.127608 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.127537    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:07.425179 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.424995    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 13:09:07.436291 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:09:07.436260    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181d5ad2_c22d_4091_b14b_38cd5c105e3f.slice/crio-542253c26f759b90a14fae7ce45623859641b7cb81629b2477d96a28adee6d08 WatchSource:0}: Error finding container 542253c26f759b90a14fae7ce45623859641b7cb81629b2477d96a28adee6d08: Status 404 returned error can't find the container with id 542253c26f759b90a14fae7ce45623859641b7cb81629b2477d96a28adee6d08
Nov 03 13:09:07.545034 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.544993    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerStarted","Data":"542253c26f759b90a14fae7ce45623859641b7cb81629b2477d96a28adee6d08"}
Nov 03 13:09:07.551858 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.551823    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8wlg" event={"ID":"67dddf6c-5f0c-4894-aeb2-c26135c32c07","Type":"ContainerStarted","Data":"4f025e387008c5adb8b97b4279bf952b4bb172ce4f7c00ef8f2c98c62e57dec9"}
Nov 03 13:09:07.682328 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.682224    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:07.834292 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:07.834263    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:08.558289 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.558247    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8wlg" event={"ID":"67dddf6c-5f0c-4894-aeb2-c26135c32c07","Type":"ContainerStarted","Data":"43978b4fc71080884d26e316bfd509ede9d9533068f4d1e1c98c8ac0201ece19"}
Nov 03 13:09:08.560789 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.560761    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdgff" event={"ID":"e1705c77-7f46-4719-8dd1-51993028ab1e","Type":"ContainerStarted","Data":"84e4b8bf9134488f170ee66d0ec5528bc8d266c7d5a51c25d99eba87e2da1e6d"}
Nov 03 13:09:08.560921 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.560815    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdgff" event={"ID":"e1705c77-7f46-4719-8dd1-51993028ab1e","Type":"ContainerStarted","Data":"29fb51c07764379fb5d920d7c5ceb7e85037e06cecbb27935026b452d2313831"}
Nov 03 13:09:08.560921 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.560844    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:08.562374 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.562348    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz" event={"ID":"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99","Type":"ContainerStarted","Data":"f80003012ee87436a400d4932d10e4d7f0d15fc1d5b98cbe864d19e63a4be706"}
Nov 03 13:09:08.562806 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.562773    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:08.569516 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.569489    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:08.581580 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.581540    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l8wlg" podStartSLOduration=1.058600335 podStartE2EDuration="3.581524094s" totalImagesPullingTime="2.522923759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:09:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:08.580003367 +0000 UTC m=+64.855439469" watchObservedRunningTime="2026-11-03 13:09:08.581524094 +0000 UTC m=+64.856960189"
Nov 03 13:09:08.602176 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.602128    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-qdgff" podStartSLOduration=1.420472219 podStartE2EDuration="3.602114375s" totalImagesPullingTime="2.181642156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:08.60175361 +0000 UTC m=+64.877189707" watchObservedRunningTime="2026-11-03 13:09:08.602114375 +0000 UTC m=+64.877550477"
Nov 03 13:09:08.623717 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:08.623663    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz" podStartSLOduration=1.662111934 podStartE2EDuration="3.623643613s" totalImagesPullingTime="1.961531679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:08.623406599 +0000 UTC m=+64.898842696" watchObservedRunningTime="2026-11-03 13:09:08.623643613 +0000 UTC m=+64.899079710"
Nov 03 13:09:09.003905 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.003344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:09.003905 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.003400    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:09.006096 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.006072    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:09.006344 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.006327    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:09.104132 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.104107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:09.106929 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.106907    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:09.167959 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.167924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:09.268793 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.268710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:09.295863 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.295162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:09.436841 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.436809    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5gj6w"]
Nov 03 13:09:09.439724 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:09:09.439693    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2aa382_9410_48f2_b139_3d9d12a26b3b.slice/crio-40a8cb115868cf4c23f9527acfc8347408aea5be584db8d22b8664596e8d8707 WatchSource:0}: Error finding container 40a8cb115868cf4c23f9527acfc8347408aea5be584db8d22b8664596e8d8707: Status 404 returned error can't find the container with id 40a8cb115868cf4c23f9527acfc8347408aea5be584db8d22b8664596e8d8707
Nov 03 13:09:09.543086 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.542990    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lr88l"]
Nov 03 13:09:09.548902 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:09:09.548874    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d48d6ac_4ab0_49f7_9aa1_3cb39131e5df.slice/crio-d7f9e657001385ef56cdf359398e576ee6a631077d1f54a703908339a4ee65a7 WatchSource:0}: Error finding container d7f9e657001385ef56cdf359398e576ee6a631077d1f54a703908339a4ee65a7: Status 404 returned error can't find the container with id d7f9e657001385ef56cdf359398e576ee6a631077d1f54a703908339a4ee65a7
Nov 03 13:09:09.566587 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.566561    2576 generic.go:356] "Generic (PLEG): container finished" podID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397" exitCode=0
Nov 03 13:09:09.566920 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.566625    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerDied","Data":"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"}
Nov 03 13:09:09.567798 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.567776    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lr88l" event={"ID":"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df","Type":"ContainerStarted","Data":"d7f9e657001385ef56cdf359398e576ee6a631077d1f54a703908339a4ee65a7"}
Nov 03 13:09:09.568918 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.568900    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5gj6w" event={"ID":"0d2aa382-9410-48f2-b139-3d9d12a26b3b","Type":"ContainerStarted","Data":"40a8cb115868cf4c23f9527acfc8347408aea5be584db8d22b8664596e8d8707"}
Nov 03 13:09:09.571305 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.571286    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zfk5q" event={"ID":"c40982af-b5a4-4f58-b35e-93208bfd793e","Type":"ContainerStarted","Data":"28c8b5b52fe239ffc8515c93a9f98886204ef86e1c9c8d5afba95eed6ea64164"}
Nov 03 13:09:09.574102 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.574082    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dnqj9"]
Nov 03 13:09:09.574948 ip-10-0-139-248 kubenswrapper[2576]: W1103 13:09:09.574927    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bd042b_b415_46cc_a7af_aa4a57ed11d5.slice/crio-3867961ad65dfc22e3b7f2421e3580128c9c80b594a78f2843b63d0f2f9dd2bb WatchSource:0}: Error finding container 3867961ad65dfc22e3b7f2421e3580128c9c80b594a78f2843b63d0f2f9dd2bb: Status 404 returned error can't find the container with id 3867961ad65dfc22e3b7f2421e3580128c9c80b594a78f2843b63d0f2f9dd2bb
Nov 03 13:09:09.614553 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:09.614508    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zfk5q" podStartSLOduration=1.271946553 podStartE2EDuration="4.614496339s" totalImagesPullingTime="3.342549786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:09.614005473 +0000 UTC m=+65.889441569" watchObservedRunningTime="2026-11-03 13:09:09.614496339 +0000 UTC m=+65.889932434"
Nov 03 13:09:10.574874 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:10.574832    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dnqj9" event={"ID":"09bd042b-b415-46cc-a7af-aa4a57ed11d5","Type":"ContainerStarted","Data":"3867961ad65dfc22e3b7f2421e3580128c9c80b594a78f2843b63d0f2f9dd2bb"}
Nov 03 13:09:11.096854 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:11.096823    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:11.263603 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:11.263561    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:11.580720 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:11.580685    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5gj6w" event={"ID":"0d2aa382-9410-48f2-b139-3d9d12a26b3b","Type":"ContainerStarted","Data":"167614a9a5ba01045894240be41acf7967baf08d10a6c6481f4021aed9b1caef"}
Nov 03 13:09:11.581161 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:11.580743    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5gj6w" event={"ID":"0d2aa382-9410-48f2-b139-3d9d12a26b3b","Type":"ContainerStarted","Data":"ab120687b6b7f891b279a60ca6616bad4dfcda3bf7937ecddfc97abcdefbd24d"}
Nov 03 13:09:11.605195 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:11.605127    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5gj6w" podStartSLOduration=66.529531501 podStartE2EDuration="1m7.605108263s" totalImagesPullingTime="1.075576762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:11.602892306 +0000 UTC m=+67.878328403" watchObservedRunningTime="2026-11-03 13:09:11.605108263 +0000 UTC m=+67.880544361"
Nov 03 13:09:12.848508 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:12.848467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:13.589660 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:13.589619    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerStarted","Data":"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"}
Nov 03 13:09:13.589859 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:13.589731    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerStarted","Data":"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"}
Nov 03 13:09:14.654580 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:14.654546    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:14.785987 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:14.785915    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:15.655575 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:15.655542    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:15.795901 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:15.795875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:16.308452 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:16.308415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:16.480167 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:16.480133    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:16.605279 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:16.604399    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerStarted","Data":"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"}
Nov 03 13:09:16.605279 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:16.604475    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerStarted","Data":"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"}
Nov 03 13:09:16.605279 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:16.604515    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerStarted","Data":"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"}
Nov 03 13:09:16.605279 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:16.604558    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerStarted","Data":"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"}
Nov 03 13:09:16.634002 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:16.633949    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.35047982 podStartE2EDuration="10.633933802s" totalImagesPullingTime="7.264041629s" totalInitContainerRuntime="19.412353ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:16.632381094 +0000 UTC m=+72.907817201" watchObservedRunningTime="2026-11-03 13:09:16.633933802 +0000 UTC m=+72.909369908"
Nov 03 13:09:16.841701 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:16.841660    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:16.981905 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:16.981870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:17.128089 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:17.128055    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:18.292071 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:18.292035    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:18.433460 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:18.433429    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:18.574344 ip-10-0-139-248 kubenswrapper[2576]: I1103 13:09:18.574273    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:18.856510 ip-10-0-139-248 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 13:09:18.871688 ip-10-0-139-248 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 13:09:18.871915 ip-10-0-139-248 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 13:09:18.872162 ip-10-0-139-248 systemd[1]: kubelet.service: Consumed 3.938s CPU time.
Nov 03 13:09:18.890723 ip-10-0-139-248 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 13:09:18.982619 ip-10-0-139-248 kubenswrapper[5420]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:09:18.982619 ip-10-0-139-248 kubenswrapper[5420]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 13:09:18.982619 ip-10-0-139-248 kubenswrapper[5420]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:09:18.982619 ip-10-0-139-248 kubenswrapper[5420]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 13:09:18.985016 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.984921    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:09:18.985016 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985012    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:09:18.985016 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985019    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:09:18.985016 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985023    5420 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985028    5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985032    5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985036    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985040    5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985044    5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985048    5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985052    5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985055    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985060    5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985068    5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985073    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985076    5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985080    5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985084    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985088    5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985092    5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985096    5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985102    5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:09:18.985379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985108    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985112    5420 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985119    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985122    5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985126    5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985130    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985134    5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985138    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985142    5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985146    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985150    5420 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985154    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985158    5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985162    5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985167    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985172    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985177    5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985181    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985185    5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985190    5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:09:18.986196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985194    5420 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985199    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985203    5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985229    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985234    5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985238    5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985242    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985246    5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985250    5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985255    5420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985260    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985265    5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985269    5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985273    5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985278    5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985282    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985286    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985290    5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985295    5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985299    5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:09:18.987080 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985303    5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985308    5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985312    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985316    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985320    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985325    5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985330    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985335    5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985340    5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985345    5420 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985352    5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985357    5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985364    5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985370    5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985375    5420 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985380    5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985384    5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985388    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985392    5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:09:18.987970 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985396    5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985401    5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985405    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985409    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985413    5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985417    5420 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985422    5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985426    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985431    5420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985435    5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985441    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985446    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985450    5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985454    5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985459    5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985464    5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985468    5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985473    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985482    5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985487    5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:09:18.988821 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985492    5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985496    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985501    5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985505    5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.985511    5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986786    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986798    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986805    5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986825    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986830    5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986835    5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986840    5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986844    5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986848    5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986852    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986857    5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986862    5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986866    5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986870    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:09:18.989732 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986875    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986879    5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986883    5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986887    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986892    5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986896    5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986900    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986904    5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986909    5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986913    5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986917    5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986922    5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986926    5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986931    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986935    5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986939    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986944    5420 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986948    5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986960    5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986965    5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:09:18.990606 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986969    5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986974    5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986978    5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986983    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986987    5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986991    5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.986995    5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987000    5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987004    5420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987009    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987013    5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987017    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987022    5420 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987026    5420 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987031    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987035    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987039    5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987043    5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987048    5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987052    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:09:18.991458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987056    5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987060    5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987064    5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987069    5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987073    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987079    5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987084    5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987089    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987095    5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987102    5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987109    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987115    5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987119    5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987124    5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987128    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987132    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987137    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987141    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987145    5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:09:18.992013 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987150    5420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987155    5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987159    5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987163    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987167    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987172    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987176    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987181    5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987186    5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987190    5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987194    5420 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987198    5420 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987203    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987232    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987238    5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987243    5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987248    5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987252    5420 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987256    5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987261    5420 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:09:18.992553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987265    5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987269    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987273    5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987277    5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987281    5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987286    5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987296    5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987301    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987305    5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987309    5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987313    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987318    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.987322    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988140    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988150    5420 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988154    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988159    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988163    5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988167    5420 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:09:18.993044 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988172    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988177    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988181    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988186    5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988190    5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988194    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988198    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988202    5420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988225    5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988230    5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988235    5420 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988239    5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988244    5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988248    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988252    5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988257    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988261    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988265    5420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988271    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988275    5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:09:18.993578 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988279    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988284    5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988288    5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988292    5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988297    5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988301    5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988305    5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988309    5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988313    5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988318    5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988322    5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988326    5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988331    5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988335    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988339    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988344    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988348    5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988352    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988356    5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:09:18.994120 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988361    5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988365    5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988370    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988374    5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988379    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988383    5420 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988387    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988391    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988395    5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988400    5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988404    5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988408    5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988413    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988418    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988422    5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988426    5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988430    5420 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988435    5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988443    5420 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988447    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:09:18.994610 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988451    5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988456    5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988460    5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988464    5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988468    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988472    5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988477    5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988481    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988485    5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988489    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988493    5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988497    5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988501    5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988506    5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988510    5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988514    5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988518    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988528    5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988534    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:09:18.995111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988538    5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988542    5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988549    5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988555    5420 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988560    5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988565    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988570    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988574    5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988579    5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988583    5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988587    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988593    5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988598    5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988603    5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988607    5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988611    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988615    5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988620    5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988624    5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:09:18.995645 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988628    5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988632    5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.988636    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988772    5420 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988789    5420 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988799    5420 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988806    5420 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988813    5420 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988819    5420 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988825    5420 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988831    5420 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988839    5420 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988844    5420 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988850    5420 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988856    5420 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988862    5420 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988867    5420 flags.go:64] FLAG: --cgroup-root=""
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988872    5420 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988877    5420 flags.go:64] FLAG: --client-ca-file=""
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988881    5420 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988887    5420 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988893    5420 flags.go:64] FLAG: --cluster-domain=""
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988898    5420 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 13:09:18.996311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988903    5420 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988911    5420 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988916    5420 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988923    5420 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988928    5420 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988933    5420 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988938    5420 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988943    5420 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988948    5420 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988953    5420 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988958    5420 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988963    5420 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988970    5420 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988975    5420 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988980    5420 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988985    5420 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988990    5420 flags.go:64] FLAG: --enable-server="true"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.988995    5420 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989001    5420 flags.go:64] FLAG: --event-burst="100"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989006    5420 flags.go:64] FLAG: --event-qps="50"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989013    5420 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989018    5420 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989023    5420 flags.go:64] FLAG: --eviction-hard=""
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989029    5420 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 13:09:18.997062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989034    5420 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989040    5420 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989045    5420 flags.go:64] FLAG: --eviction-soft=""
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989050    5420 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989054    5420 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989059    5420 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989065    5420 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989069    5420 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989074    5420 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989081    5420 flags.go:64] FLAG: --feature-gates=""
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989088    5420 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989093    5420 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989098    5420 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989103    5420 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989108    5420 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989113    5420 flags.go:64] FLAG: --help="false"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989118    5420 flags.go:64] FLAG: --hostname-override="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989123    5420 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989128    5420 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989133    5420 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989139    5420 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989145    5420 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989149    5420 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 13:09:18.997701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989154    5420 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989158    5420 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989163    5420 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989168    5420 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989174    5420 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989180    5420 flags.go:64] FLAG: --kube-reserved=""
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989185    5420 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989190    5420 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989195    5420 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989200    5420 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989204    5420 flags.go:64] FLAG: --lock-file=""
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989224    5420 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989229    5420 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989234    5420 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989243    5420 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989248    5420 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989254    5420 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989259    5420 flags.go:64] FLAG: --logging-format="text"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989266    5420 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989271    5420 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989276    5420 flags.go:64] FLAG: --manifest-url=""
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989281    5420 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989288    5420 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989293    5420 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989299    5420 flags.go:64] FLAG: --max-pods="110"
Nov 03 13:09:18.998274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989304    5420 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989309    5420 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989314    5420 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989319    5420 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989324    5420 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989329    5420 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989334    5420 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989345    5420 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989350    5420 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989355    5420 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989360    5420 flags.go:64] FLAG: --pod-cidr=""
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989364    5420 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989371    5420 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989376    5420 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989381    5420 flags.go:64] FLAG: --port="10250"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989386    5420 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989391    5420 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0be33d50c0802780b"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989396    5420 flags.go:64] FLAG: --qos-reserved=""
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989401    5420 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989406    5420 flags.go:64] FLAG: --register-node="true"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989411    5420 flags.go:64] FLAG: --register-with-taints=""
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989416    5420 flags.go:64] FLAG: --registry-burst="10"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989421    5420 flags.go:64] FLAG: --registry-qps="5"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989426    5420 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989432    5420 flags.go:64] FLAG: --reserved-memory=""
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989440    5420 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 13:09:18.998900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989445    5420 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989451    5420 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989455    5420 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989460    5420 flags.go:64] FLAG: --runonce="false"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989465    5420 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989470    5420 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989475    5420 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989480    5420 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989485    5420 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989490    5420 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989495    5420 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989500    5420 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989505    5420 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989509    5420 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989514    5420 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989519    5420 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989524    5420 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989529    5420 flags.go:64] FLAG: --system-cgroups=""
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989535    5420 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989544    5420 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989549    5420 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989553    5420 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989560    5420 flags.go:64] FLAG: --tls-min-version=""
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989565    5420 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989569    5420 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 13:09:18.999598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989574    5420 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989579    5420 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989584    5420 flags.go:64] FLAG: --v="2"
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989591    5420 flags.go:64] FLAG: --version="false"
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989598    5420 flags.go:64] FLAG: --vmodule=""
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989604    5420 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.989612    5420 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989844    5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989853    5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989858    5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989863    5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989867    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989871    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989878    5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989885    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989891    5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989896    5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989901    5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989905    5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989910    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:09:19.000236 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989914    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989919    5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989923    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989927    5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989931    5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989938    5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989943    5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989948    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989952    5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989956    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989960    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989964    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989969    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989972    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989976    5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989981    5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989985    5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989989    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.989995    5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990000    5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:09:19.000737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990005    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990009    5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990013    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990017    5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990022    5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990026    5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990030    5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990035    5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990039    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990043    5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990048    5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990052    5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990056    5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990060    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990064    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990068    5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990073    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990079    5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990083    5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990088    5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:09:19.001413 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990092    5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990097    5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990101    5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990105    5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990109    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990114    5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990119    5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990123    5420 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990128    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990132    5420 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990138    5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990142    5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990148    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990152    5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990157    5420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990162    5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990166    5420 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990170    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990175    5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990179    5420 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:09:19.002004 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990183    5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990187    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990192    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990197    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990201    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990205    5420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990226    5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990231    5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990235    5420 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990241    5420 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990245    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990249    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990255    5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990259    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990263    5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990268    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990273    5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990277    5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990282    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990286    5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:09:19.002542 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990290    5420 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990294    5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990301    5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990305    5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990309    5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990314    5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990321    5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990329    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990334    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990339    5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990343    5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990347    5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990351    5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.990396    5420 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990451    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:09:19.003031 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990457    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990462    5420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990466    5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990470    5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990474    5420 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990479    5420 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990483    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990490    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990494    5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990499    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990503    5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990507    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990513    5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990517    5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990521    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990525    5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990530    5420 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990534    5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990538    5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990544    5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:09:19.003451 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990548    5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990553    5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990559    5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990566    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990571    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990576    5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990580    5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990585    5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990589    5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990593    5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990598    5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990602    5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990606    5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990610    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990614    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990620    5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990626    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990630    5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:09:19.004028 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990634    5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990644    5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990649    5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990653    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990657    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990662    5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990666    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990670    5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990674    5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990679    5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990683    5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990687    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990692    5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990696    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990701    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990705    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990709    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990713    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990718    5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990723    5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:09:19.004650 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990727    5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990731    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990736    5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990740    5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990744    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990749    5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990753    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990757    5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990761    5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990766    5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990770    5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990774    5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990778    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990785    5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990789    5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990794    5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990798    5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990802    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990807    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:09:19.005277 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990811    5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990815    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990820    5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990824    5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990828    5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990833    5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990837    5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990841    5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990848    5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990852    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990857    5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990861    5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990866    5420 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990871    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990875    5420 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990879    5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990884    5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990888    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990892    5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990896    5420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:09:19.005767 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990901    5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990905    5420 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990910    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990914    5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990918    5420 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990922    5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990929    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.990935    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.995169    5420 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.995186    5420 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.995201    5420 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.995206    5420 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995286    5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995292    5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995295    5420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995298    5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995301    5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995303    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995308    5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995311    5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:09:19.006339 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995315    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995318    5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995320    5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995323    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995326    5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995329    5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995332    5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995335    5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995343    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995346    5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995349    5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995352    5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995355    5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995358    5420 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995360    5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995363    5420 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995366    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995368    5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995371    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:09:19.006823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995375    5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995378    5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995388    5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995391    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995394    5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995396    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995399    5420 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995401    5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995404    5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995406    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995409    5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995412    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995414    5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995416    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995419    5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995421    5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995424    5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995426    5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995429    5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995432    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:09:19.007382 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995435    5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995438    5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995440    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995443    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995445    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995448    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995450    5420 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995453    5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995456    5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995458    5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995461    5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995463    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995466    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995469    5420 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995471    5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995480    5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995482    5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995485    5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995487    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:09:19.007856 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995490    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995492    5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995495    5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995497    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995500    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995502    5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995505    5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995508    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995510    5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995513    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995515    5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995518    5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995521    5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995524    5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995527    5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995529    5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995532    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995534    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995537    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995539    5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:09:19.008326 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995542    5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995544    5420 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995547    5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995549    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995552    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995554    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995557    5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995559    5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995562    5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995570    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995573    5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995575    5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995578    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995580    5420 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995583    5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995585    5420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995587    5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995590    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995592    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:09:19.008807 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.995595    5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.995623    5420 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996197    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996225    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996232    5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996238    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996242    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996248    5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996252    5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996262    5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996267    5420 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996272    5420 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996276    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996280    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996285    5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:09:19.009286 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996290    5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996295    5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996299    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996304    5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996308    5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996312    5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996317    5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996326    5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996367    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996371    5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996376    5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996380    5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996385    5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996389    5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996393    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996398    5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996403    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996407    5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996411    5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996420    5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:09:19.009656 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996425    5420 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996429    5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996433    5420 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996437    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996441    5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996445    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996449    5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996453    5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996457    5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996461    5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996465    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996469    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996480    5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996486    5420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996491    5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996495    5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996500    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996504    5420 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996508    5420 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996512    5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:09:19.010167 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996517    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996546    5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996550    5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996555    5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996559    5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996568    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996573    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996577    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996581    5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996585    5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996590    5420 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996594    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996608    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996612    5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996616    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996620    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996624    5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996634    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996638    5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996642    5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:09:19.010901 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996646    5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996650    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996655    5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996659    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996663    5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996667    5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996671    5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996675    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996680    5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996686    5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996697    5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996701    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996706    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996710    5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996739    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996744    5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996748    5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996752    5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996757    5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:09:19.011513 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996760    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996764    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996768    5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996777    5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996781    5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996854    5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996870    5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996874    5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996880    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996985    5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.996997    5420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997000    5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997003    5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997006    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.997039    5420 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997091    5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 13:09:19.012118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997097    5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997100    5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997103    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997106    5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997109    5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997111    5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997114    5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997117    5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997121    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997124    5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997126    5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997129    5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997133    5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997136    5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997138    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997141    5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997144    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997147    5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997149    5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997152    5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 13:09:19.012643 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997155    5420 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997157    5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997162    5420 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997165    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997167    5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997170    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997172    5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997176    5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997179    5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997182    5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997184    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997187    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997191    5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997195    5420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997198    5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997201    5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997203    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997206    5420 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997229    5420 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997232    5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 13:09:19.013280 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997235    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997238    5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997240    5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997243    5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997246    5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997248    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997251    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997253    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997256    5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997259    5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997261    5420 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997265    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997268    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997271    5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997275    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997277    5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997280    5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997283    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997285    5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997289    5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 13:09:19.014006 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997292    5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997294    5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997298    5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997300    5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997303    5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997305    5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997308    5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997310    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997313    5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997317    5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997321    5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997324    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997328    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997330    5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997334    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997337    5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997340    5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997343    5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997346    5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 13:09:19.014607 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997348    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997351    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997354    5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997356    5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997359    5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997361    5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997364    5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997367    5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997370    5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997373    5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997377    5420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997380    5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997384    5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997386    5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997389    5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997392    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997394    5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997397    5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997400    5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 13:09:19.015525 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997402    5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997405    5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997408    5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997410    5420 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997413    5420 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997415    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:18.997418    5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.997473    5420 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.998452    5420 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.998505    5420 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.998679    5420 server.go:1014] "Starting client certificate rotation"
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.998827    5420 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.998887    5420 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 13:03:04 +0000 UTC" deadline="2028-04-25 06:28:44.988212065 +0000 UTC"
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.998933    5420 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12929h19m25.989282272s"
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:18.999224    5420 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.000365    5420 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 13:09:19.016302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.001932    5420 log.go:25] "Validated CRI v1 runtime API"
Nov 03 13:09:19.016923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.003684    5420 log.go:25] "Validated CRI v1 image API"
Nov 03 13:09:19.016923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.004478    5420 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 13:09:19.016923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.006389    5420 fs.go:135] Filesystem UUIDs: map[3fdab71c-10fc-409f-9c55-acb6cf004c6e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dcffbb01-f8e8-45d6-b39f-765a512536f5:/dev/nvme0n1p3]
Nov 03 13:09:19.017107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.006405    5420 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a61ef30690bfe69177dbda8ce018d0180dca73085a4be9b5826c651d34a2ae4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a61ef30690bfe69177dbda8ce018d0180dca73085a4be9b5826c651d34a2ae4/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1eb9f33307b58153cf0ce3ab50bac2a1e7e5004fc33cb2a6d87f064107d73d2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1eb9f33307b58153cf0ce3ab50bac2a1e7e5004fc33cb2a6d87f064107d73d2e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24900b96e5aa02946a235af3656ad6fe62d6cad2008ca8569568f6034cd5d130/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24900b96e5aa02946a235af3656ad6fe62d6cad2008ca8569568f6034cd5d130/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/279e8c90c9de5e2c3911a2aac1346cae7ffe731d3e564ccaf0cd915f068cf39a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/279e8c90c9de5e2c3911a2aac1346cae7ffe731d3e564ccaf0cd915f068cf39a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3867961ad65dfc22e3b7f2421e3580128c9c80b594a78f2843b63d0f2f9dd2bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3867961ad65dfc22e3b7f2421e3580128c9c80b594a78f2843b63d0f2f9dd2bb/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ad193723cf185eca2af116534b69a27edc90f9ec878f7f94aa0d133122c5847/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ad193723cf185eca2af116534b69a27edc90f9ec878f7f94aa0d133122c5847/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f46bdfe18ad7bd52c491497087779518ca247162bfeb9adca48933c258e29c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f46bdfe18ad7bd52c491497087779518ca247162bfeb9adca48933c258e29c5/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40a8cb115868cf4c23f9527acfc8347408aea5be584db8d22b8664596e8d8707/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40a8cb115868cf4c23f9527acfc8347408aea5be584db8d22b8664596e8d8707/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51c9df65e99c8e774fe7562ff97c0be38c8d184bc9565eee4dad5b27368d9c61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51c9df65e99c8e774fe7562ff97c0be38c8d184bc9565eee4dad5b27368d9c61/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/542253c26f759b90a14fae7ce45623859641b7cb81629b2477d96a28adee6d08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/542253c26f759b90a14fae7ce45623859641b7cb81629b2477d96a28adee6d08/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b01e044c6f2cf78b6d00062244c51da4915a3e081f4ac5cd50863f59cb9b8c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b01e044c6f2cf78b6d00062244c51da4915a3e081f4ac5cd50863f59cb9b8c8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5dc2b06e68791904bccd9bd8df8e3adf716a720cd98d047704d55b78877e53e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5dc2b06e68791904bccd9bd8df8e3adf716a720cd98d047704d55b78877e53e3/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b65dadc004a34cdc6bcc15e20663bc5921b5e4bcd3b5cd163e09a289dc9c941/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b65dadc004a34cdc6bcc15e20663bc5921b5e4bcd3b5cd163e09a289dc9c941/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a90b2cfd4cb781a986a7e3cb4b975dc644532151aa3dd9c8fb3202fd8a5f0b49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a90b2cfd4cb781a986a7e3cb4b975dc644532151aa3dd9c8fb3202fd8a5f0b49/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3028bcd6fdccb591f9779c911708f5c164c1563ceb1607361d7731bd3840f05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3028bcd6fdccb591f9779c911708f5c164c1563ceb1607361d7731bd3840f05/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7f9e657001385ef56cdf359398e576ee6a631077d1f54a703908339a4ee65a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7f9e657001385ef56cdf359398e576ee6a631077d1f54a703908339a4ee65a7/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd372399011085a16130308c4f082f633364c6622744300442628c4bf557c220/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd372399011085a16130308c4f082f633364c6622744300442628c4bf557c220/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebbe8aca8402f1038ae4c63f32498a1bb680533d8d15134bf6160b68ab7f08b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebbe8aca8402f1038ae4c63f32498a1bb680533d8d15134bf6160b68ab7f08b4/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f950a5f26311d75c891f5a30bac42e0715df627a3c435b5aa890ff02f0f747c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f950a5f26311d75c891f5a30bac42e0715df627a3c435b5aa890ff02f0f747c5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09bd042b-b415-46cc-a7af-aa4a57ed11d5/volumes/kubernetes.io~projected/kube-api-access-ww4nl:{mountpoint:/var/lib/kubelet/pods/09bd042b-b415-46cc-a7af-aa4a57ed11d5/volumes/kubernetes.io~projected/kube-api-access-ww4nl major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d2aa382-9410-48f2-b139-3d9d12a26b3b/volumes/kubernetes.io~projected/kube-api-access-rfrxp:{mountpoint:/var/lib/kubelet/pods/0d2aa382-9410-48f2-b139-3d9d12a26b3b/volumes/kubernetes.io~projected/kube-api-access-rfrxp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d2aa382-9410-48f2-b139-3d9d12a26b3b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0d2aa382-9410-48f2-b139-3d9d12a26b3b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f20bcca-4ad5-4241-bf25-c59211b8eb9c/volumes/kubernetes.io~projected/kube-api-access-st6rh:{mountpoint:/var/lib/kubelet/pods/0f20bcca-4ad5-4241-bf25-c59211b8eb9c/volumes/kubernetes.io~projected/kube-api-access-st6rh major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f20bcca-4ad5-4241-bf25-c59211b8eb9c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0f20bcca-4ad5-4241-bf25-c59211b8eb9c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f20bcca-4ad5-4241-bf25-c59211b8eb9c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/0f20bcca-4ad5-4241-bf25-c59211b8eb9c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:112 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volume-subpaths/web-config/config-reloader/2 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volume-subpaths/web-config/prometheus/15 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~projected/kube-api-access-7r5cj:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~projected/kube-api-access-7r5cj major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~projected/tls-assets major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/web-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01/volumes/kubernetes.io~projected/kube-api-access-46pvl:{mountpoint:/var/lib/kubelet/pods/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01/volumes/kubernetes.io~projected/kube-api-access-46pvl major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22aec86e-8e32-4e9b-82d7-218a2ceb3926/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/22aec86e-8e32-4e9b-82d7-218a2ceb3926/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642/volumes/kubernetes.io~projected/kube-api-access-d94tq:{mountpoint:/var/lib/kubelet/pods/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642/volumes/kubernetes.io~projected/kube-api-access-d94tq major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3acab04e-53c1-4ceb-b571-246d23c55934/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3acab04e-53c1-4ceb-b571-246d23c55934/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3acab04e-53c1-4ceb-b571-246d23c55934/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3acab04e-53c1-4ceb-b571-246d23c55934/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3acab04e-53c1-4ceb-b571-246d23c55934/volumes/kubernetes.io~projected/kube-api-access-j48g5:{mountpoint:/var/lib/kubelet/pods/3acab04e-53c1-4ceb-b571-246d23c55934/volumes/kubernetes.io~projected/kube-api-access-j48g5 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99/volumes/kubernetes.io~projected/kube-api-access-x7zln:{mountpoint:/var/lib/kubelet/pods/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99/volumes/kubernetes.io~projected/kube-api-access-x7zln major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67dddf6c-5f0c-4894-aeb2-c26135c32c07/volumes/kubernetes.io~projected/kube-api-access-m6x6m:{mountpoint:/var/lib/kubelet/pods/67dddf6c-5f0c-4894-aeb2-c26135c32c07/volumes/kubernetes.io~projected/kube-api-access-m6x6m major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67dddf6c-5f0c-4894-aeb2-c26135c32c07/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/67dddf6c-5f0c-4894-aeb2-c26135c32c07/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8db0e985-d36d-485d-91ae-917f751a61a1/volumes/kubernetes.io~projected/kube-api-access-jvvx7:{mountpoint:/var/lib/kubelet/pods/8db0e985-d36d-485d-91ae-917f751a61a1/volumes/kubernetes.io~projected/kube-api-access-jvvx7 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22ec490-f623-4cda-a693-e6669a1a3324/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a22ec490-f623-4cda-a693-e6669a1a3324/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22ec490-f623-4cda-a693-e6669a1a3324/volumes/kubernetes.io~projected/kube-api-access-t9bt2:{mountpoint:/var/lib/kubelet/pods/a22ec490-f623-4cda-a693-e6669a1a3324/volumes/kubernetes.io~projected/kube-api-access-t9bt2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22ec490-f623-4cda-a693-e6669a1a3324/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a22ec490-f623-4cda-a693-e6669a1a3324/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa3f9f6e-77c4-4053-9289-65469d56d6ac/volumes/kubernetes.io~projected/kube-api-access-zfm9p:{mountpoint:/var/lib/kubelet/pods/aa3f9f6e-77c4-4053-9289-65469d56d6ac/volumes/kubernetes.io~projected/kube-api-access-zfm9p major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c40982af-b5a4-4f58-b35e-93208bfd793e/volumes/kubernetes.io~projected/kube-api-access-dw4ml:{mountpoint:/var/lib/kubelet/pods/c40982af-b5a4-4f58-b35e-93208bfd793e/volumes/kubernetes.io~projected/kube-api-access-dw4ml major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c40982af-b5a4-4f58-b35e-93208bfd793e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c40982af-b5a4-4f58-b35e-93208bfd793e/volumes/kubernetes.io~secret/cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6ccec32-8bf2-4f47-8d30-abd99b513d1b/volumes/kubernetes.io~projected/kube-api-access-xfr92:{mountpoint:/var/lib/kubelet/pods/c6ccec32-8bf2-4f47-8d30-abd99b513d1b/volumes/kubernetes.io~projected/kube-api-access-xfr92 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dca216a1-acaf-4fe6-b833-4fc7e25ad427/volumes/kubernetes.io~projected/kube-api-access-4g7xv:{mountpoint:/var/lib/kubelet/pods/dca216a1-acaf-4fe6-b833-4fc7e25ad427/volumes/kubernetes.io~projected/kube-api-access-4g7xv major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1705c77-7f46-4719-8dd1-51993028ab1e/volumes/kubernetes.io~projected/kube-api-access-jh2mh:{mountpoint:/var/lib/kubelet/pods/e1705c77-7f46-4719-8dd1-51993028ab1e/volumes/kubernetes.io~projected/kube-api-access-jh2mh major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1705c77-7f46-4719-8dd1-51993028ab1e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/e1705c77-7f46-4719-8dd1-51993028ab1e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:213 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f13fdadd8203599ddd207184c53f8a361d96cb5f8ee46d2133081369ae3bf1c1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1909d4d03f11e355784c5646e8eb69d87fc8d3dc1a0fde459fc608754d2cee73/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/53de3ceb8ff532467e06f2672498512d11eb5b7a07c0b8095f31f607dd018c4c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/b6e4bbf1e9e0e9e6a799a9cea49a50f2dd523833f6ddf30e9f16a3da9457ca88/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/fd00a3e81adf1f8f152a7678cd4d8d8a56cb6635cf31647062b064dfeeade91d/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/d8bcee180c4d65c8e8d5e55aae04023e0c53d59483737f9a81738e9c778f41a7/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ed08b49d6cfb8bbe7cc5d73be499ff58b4cacd5278d434adf9aa51e8916cf2a5/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/d185270528344fb15b0e91418240fc304c266310689ccd2ad0b43fcbb2d22c28/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/c6d9a095d577235805d15da8ec78f906540ea62f67d7cd4f7c7f8adda66524d1/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/bf47650a660800e00e309403136e008975a8c58741a80039501dc553804e7de1/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/b3a91c882229bf2104634134772c6f682ee20adf68fb9e1441b214970de9f916/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/87855a9aeae05a5045c32abd7d4d9d8c9e7dc0d4e15534d65b06a155d057bad4/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/2b9f3235aed90e20253a99f26dddb39a2741a9469c621590dcc416ff862aa053/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/2606236b1e3e990a47c0975d6f9589e215c0eae5d111b484bea52a244fa248f8/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/c5fcaf749b3414d006e5fb8e088fb986736ee31425789419cb2a0d5b2653a45d/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/cea923e708913a99f96d368f755c276660713a8cd6aaaa534af21e3d3c7f1c7e/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/6625c57093f457cec7e4250d3441fba2e8a014e93ff3005efecad2be6bad48ef/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/90f4a8b356d9cc97f1a71a6cdcd6a5126db3f7602de563987545881ecb8882d7/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/fc2c28111d669cbb758b20cda98e03f6ad3764be745d909cf1e665af5b2f399d/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/b28abb4a123993b18fdae0c32a436f9a2dbb78cee71ec8e7f6079c94186ad697/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/217e2ec9f33d73194521cb222c78aa4c53f09ff9ce9e761d9520a74382f2adc1/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/3c1bb762115646ff518e522a0e313c3c57d6ea3af237435eb289c43d4ecb559a/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/81fb4d9d557b24876cf938653e8027ea8eda4f8829a0f5fd4de45b69d9d049b3/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/48568edbe92feacc8b0fdb3d2a593ee34b032857051954331fc5f1c0c80d5ce5/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/91e7fcf44b2bb2f3a734388d2c332a334ec7311bf9dcdd688ee67ebc0fa5fb9f/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/6e9316ff07b966dcf2c8e943991a7ba7c038ff386bc163a52778d67bb79ba175/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/2644f57b46546c0c81b6c66cd33d31717d1f0b449a306841ba369fd97ceb06d7/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/b64e4684b7e3360ebfc0a7f6df6db3eaa2fc0908fac59653acaba309cd0ddc71/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/c7a400271b423435a98711d2241156866343fe4041632d8ea84786b4dec8aa58/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/3afb44f7ab95c80086500abc6b217904d39efe0733c2b6ed4458f5dcff6d252b/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/71e41bc4a95a2c0bfcc46d331fc625cd44f3e0c5411f22719b649f7996591736/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/516233243a0500ec186d0ff7bf20b501672feaa88101f2dde099684ede2d5e1a/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/6f49afd38130a1eceb889534eb499693b95ce486c17bbb52aae19549825074d1/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/f40434e7b50a067ceb21571c7940fba4e77471061b33c2e29c84058a1c8ecfaf/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/6e5c033b05f8cab517a99383d687457b874dfde4d9506490f2411a49fcf0ebd8/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/19c518416c8d289faf0f616f1c2e6d691bb7f58f9fae1da71906fa6368653561/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/d45774a13aa53ddd9156b56d16c9ca1f6ef98b8b884dc1a848c9a7c091801561/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/5d427251b57e999593228a8782e092c8768678decc90a32e34510eab50a2af81/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/dee8060b2c59467b65eca18ce1fb7ef5529b985688fb01369771670ecbed70a6/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/3c96633859ee8a2bf55f1423b5e6f48ef2957bbbe030ac4ddd1791a48bb93a1f/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/b2764c8dd2934f66900ff020caa42cc85a098d3a3b38d22b0f4d41857f57b067/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/ce60fc6edd38f6e593ccabb608c8582b6fd8967b3d1e3ce2066c35f1675482be/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/88b13792ec9165833ef1269cdf4e098091ef24a9184ef3816d40754f1dfc6e4a/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/4615ce0043b2543366a3ba182783de0d20113960509fb05f1a929f974449075a/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/cd3a1cf4a087d01472ab9cfa9a309594b3f217621ba0d49d66a159b2f4cc25ff/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/c1369d9fe16ffe9a29327d50807260a616e857585cd83fd3b065f8c1b612096e/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/d97fff58b1804a6dacb4765fa35674ecd4c427f1beb695a9c096d770d02cf8d8/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/2614c32af6665dee68cc36c7128e145f7dc05e7d9b0d0dc1b02835fe91801959/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/2fa21354fa3897cba5b603a9500c8af8afefaf6173bc6cfa812dd4e8d610b179/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/119db3473bbff3070f89d69bcb4619432b8366d68d48880ae2a1b9ae64a39035/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/309b74579ee005cf3d6a240281d64ff5a234d50a91bdec3bb07592d3f22bc9fd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/81db2c0ba34d2345b52d565ae834185017e0e325d146ff2d58f09838da3300ee/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/97b9592f52627c5c1c31b1eb9c16aca0a54bd7b92ea81f598a4fe0870273f676/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/19a44e7647415a62d49cb1c29a72263ed13984f97ccc9e5d716a51539219d9f9/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/2c4497e5a07c1599e6d0ebdd18925467dc5ae272ed0ee8f6189c5aba0ffab756/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/dbde501f553fd92a7848a18c528c344eae70be0c4b5e484a8089e7b5ea1e2761/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a9fe590f530e225ef309c295151378ef53b5a0fa88b565deaf9c9f89d6f1b0d6/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f106ad91819fb5f58654102644a873b2314c974c84ddf568c505d2fafba11715/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.015730    5420 manager.go:217] Machine: {Timestamp:2026-11-03 13:09:19.014928044 +0000 UTC m=+0.092693062 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099760 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2601d6dc49d09845c1e148b68030d4 SystemUUID:ec2601d6-dc49-d098-45c1-e148b68030d4 BootID:c74f5349-78af-46dc-a695-4c70919b0570 Filesystems:[{Device:/run/containers/storage/overlay-containers/3f46bdfe18ad7bd52c491497087779518ca247162bfeb9adca48933c258e29c5/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:248 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3028bcd6fdccb591f9779c911708f5c164c1563ceb1607361d7731bd3840f05/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f20bcca-4ad5-4241-bf25-c59211b8eb9c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:200 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/0f20bcca-4ad5-4241-bf25-c59211b8eb9c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:112 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99/volumes/kubernetes.io~projected/kube-api-access-x7zln DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a90b2cfd4cb781a986a7e3cb4b975dc644532151aa3dd9c8fb3202fd8a5f0b49/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c40982af-b5a4-4f58-b35e-93208bfd793e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c6ccec32-8bf2-4f47-8d30-abd99b513d1b/volumes/kubernetes.io~projected/kube-api-access-xfr92 DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a61ef30690bfe69177dbda8ce018d0180dca73085a4be9b5826c651d34a2ae4/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51c9df65e99c8e774fe7562ff97c0be38c8d184bc9565eee4dad5b27368d9c61/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebbe8aca8402f1038ae4c63f32498a1bb680533d8d15134bf6160b68ab7f08b4/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c40982af-b5a4-4f58-b35e-93208bfd793e/volumes/kubernetes.io~projected/kube-api-access-dw4ml DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0d2aa382-9410-48f2-b139-3d9d12a26b3b/volumes/kubernetes.io~projected/kube-api-access-rfrxp DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa3f9f6e-77c4-4053-9289-65469d56d6ac/volumes/kubernetes.io~projected/kube-api-access-zfm9p DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:260 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~projected/kube-api-access-7r5cj DeviceMajor:0 DeviceMinor:262 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40a8cb115868cf4c23f9527acfc8347408aea5be584db8d22b8664596e8d8707/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7f9e657001385ef56cdf359398e576ee6a631077d1f54a703908339a4ee65a7/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01/volumes/kubernetes.io~projected/kube-api-access-46pvl DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5dc2b06e68791904bccd9bd8df8e3adf716a720cd98d047704d55b78877e53e3/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:248 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3acab04e-53c1-4ceb-b571-246d23c55934/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24900b96e5aa02946a235af3656ad6fe62d6cad2008ca8569568f6034cd5d130/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f20bcca-4ad5-4241-bf25-c59211b8eb9c/volumes/kubernetes.io~projected/kube-api-access-st6rh DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e1705c77-7f46-4719-8dd1-51993028ab1e/volumes/kubernetes.io~projected/kube-api-access-jh2mh DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:254 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/542253c26f759b90a14fae7ce45623859641b7cb81629b2477d96a28adee6d08/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09bd042b-b415-46cc-a7af-aa4a57ed11d5/volumes/kubernetes.io~projected/kube-api-access-ww4nl DeviceMajor:0 DeviceMinor:294 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1eb9f33307b58153cf0ce3ab50bac2a1e7e5004fc33cb2a6d87f064107d73d2e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67dddf6c-5f0c-4894-aeb2-c26135c32c07/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642/volumes/kubernetes.io~projected/kube-api-access-d94tq DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/279e8c90c9de5e2c3911a2aac1346cae7ffe731d3e564ccaf0cd915f068cf39a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22aec86e-8e32-4e9b-82d7-218a2ceb3926/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3867961ad65dfc22e3b7f2421e3580128c9c80b594a78f2843b63d0f2f9dd2bb/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d2aa382-9410-48f2-b139-3d9d12a26b3b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8db0e985-d36d-485d-91ae-917f751a61a1/volumes/kubernetes.io~projected/kube-api-access-jvvx7 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a22ec490-f623-4cda-a693-e6669a1a3324/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b65dadc004a34cdc6bcc15e20663bc5921b5e4bcd3b5cd163e09a289dc9c941/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e1705c77-7f46-4719-8dd1-51993028ab1e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3acab04e-53c1-4ceb-b571-246d23c55934/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b01e044c6f2cf78b6d00062244c51da4915a3e081f4ac5cd50863f59cb9b8c8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ad193723cf185eca2af116534b69a27edc90f9ec878f7f94aa0d133122c5847/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:248 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:261 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:248 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dca216a1-acaf-4fe6-b833-4fc7e25ad427/volumes/kubernetes.io~projected/kube-api-access-4g7xv DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a22ec490-f623-4cda-a693-e6669a1a3324/volumes/kubernetes.io~projected/kube-api-access-t9bt2 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd372399011085a16130308c4f082f633364c6622744300442628c4bf557c220/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:255 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a22ec490-f623-4cda-a693-e6669a1a3324/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3acab04e-53c1-4ceb-b571-246d23c55934/volumes/kubernetes.io~projected/kube-api-access-j48g5 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67dddf6c-5f0c-4894-aeb2-c26135c32c07/volumes/kubernetes.io~projected/kube-api-access-m6x6m DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:256 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f950a5f26311d75c891f5a30bac42e0715df627a3c435b5aa890ff02f0f747c5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3867961ad65dfc2 MacAddress:a2:9a:c3:7f:e1:4b Speed:10000 Mtu:8901} {Name:3f46bdfe18ad7bd MacAddress:8e:6a:7f:6b:9b:7a Speed:10000 Mtu:8901} {Name:40a8cb115868cf4 MacAddress:82:51:bd:a6:f9:e7 Speed:10000 Mtu:8901} {Name:542253c26f759b9 MacAddress:7a:34:ff:0b:3e:44 Speed:10000 Mtu:8901} {Name:a90b2cfd4cb781a MacAddress:aa:ca:b4:c3:29:7f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d0:52:1f:cf Speed:0 Mtu:9001} {Name:br-int MacAddress:b6:0d:c2:ef:4e:8b Speed:0 Mtu:8901} {Name:d7f9e657001385e MacAddress:92:67:71:65:fc:66 Speed:10000 Mtu:8901} {Name:dd372399011085a MacAddress:e2:f8:8b:48:9c:3b Speed:10000 Mtu:8901} {Name:ebbe8aca8402f10 MacAddress:c6:e9:0d:51:b3:87 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d0:52:1f:cf Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ba:de:70:af:03:b8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3e:80:0d:1a:f9:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.016383    5420 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.016439    5420 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.016775    5420 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.016805    5420 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-248.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017002    5420 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017014    5420 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017029    5420 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017056    5420 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017174    5420 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 13:09:19.017479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017475    5420 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 13:09:19.018202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017630    5420 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 13:09:19.018202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017673    5420 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 13:09:19.018202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017687    5420 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 13:09:19.018202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017699    5420 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 13:09:19.018202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017714    5420 kubelet.go:405] "Adding apiserver pod source"
Nov 03 13:09:19.018202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.017724    5420 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 13:09:19.018533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.018518    5420 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 13:09:19.018583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.018534    5420 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 13:09:19.019077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.019049    5420 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 13:09:19.032306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.032265    5420 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:09:19.032962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.032949    5420 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 13:09:19.032962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.032963    5420 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 13:09:19.033077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033068    5420 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 13:09:19.033287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033275    5420 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 13:09:19.033320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033305    5420 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 13:09:19.033449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033436    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 13:09:19.033485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033462    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 13:09:19.033485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033475    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 13:09:19.033551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033488    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 13:09:19.033551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033500    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 13:09:19.033551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033512    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 13:09:19.033551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033524    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 13:09:19.033551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033536    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 13:09:19.033551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033551    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 13:09:19.033746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033565    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 13:09:19.033746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033590    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 13:09:19.033746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033608    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 13:09:19.033746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033661    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 13:09:19.033746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.033674    5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 13:09:19.035916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.035898    5420 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:09:19.037196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.037181    5420 server.go:1263] "Started kubelet"
Nov 03 13:09:19.037308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.037254    5420 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 13:09:19.038788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.037295    5420 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 13:09:19.038788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.037374    5420 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 13:09:19.038788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.037692    5420 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 13:09:19.037773 ip-10-0-139-248 systemd[1]: Started Kubernetes Kubelet.
Nov 03 13:09:19.040924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.040015    5420 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 13:09:19.043267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.043245    5420 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 13:09:19.043267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.043259    5420 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 13:09:19.043429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.043334    5420 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 13:03:05 +0000 UTC" deadline="2028-06-26 07:42:33.340445567 +0000 UTC"
Nov 03 13:09:19.043429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.043371    5420 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14418h33m14.297095805s"
Nov 03 13:09:19.043429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.043418    5420 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 13:09:19.043569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.043427    5420 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 13:09:19.043623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.043577    5420 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 13:09:19.045779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045727    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f20bcca-4ad5-4241-bf25-c59211b8eb9c" volumeName="kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 13:09:19.045873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045781    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 13:09:19.045873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045798    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 13:09:19.045873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045811    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 13:09:19.045873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045824    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 13:09:19.045873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045836    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22aec86e-8e32-4e9b-82d7-218a2ceb3926" volumeName="kubernetes.io/configmap/22aec86e-8e32-4e9b-82d7-218a2ceb3926-konnectivity-ca" seLinuxMountContext=""
Nov 03 13:09:19.045873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045849    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67dddf6c-5f0c-4894-aeb2-c26135c32c07" volumeName="kubernetes.io/projected/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-api-access-m6x6m" seLinuxMountContext=""
Nov 03 13:09:19.045873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045862    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa3f9f6e-77c4-4053-9289-65469d56d6ac" volumeName="kubernetes.io/projected/aa3f9f6e-77c4-4053-9289-65469d56d6ac-kube-api-access-zfm9p" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045877    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045889    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045901    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40982af-b5a4-4f58-b35e-93208bfd793e" volumeName="kubernetes.io/secret/c40982af-b5a4-4f58-b35e-93208bfd793e-cert" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045924    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dca216a1-acaf-4fe6-b833-4fc7e25ad427" volumeName="kubernetes.io/projected/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kube-api-access-4g7xv" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045935    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09bd042b-b415-46cc-a7af-aa4a57ed11d5" volumeName="kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045947    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f20bcca-4ad5-4241-bf25-c59211b8eb9c" volumeName="kubernetes.io/empty-dir/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-textfile" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045966    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045971    5420 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045980    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d2aa382-9410-48f2-b139-3d9d12a26b3b" volumeName="kubernetes.io/projected/0d2aa382-9410-48f2-b139-3d9d12a26b3b-kube-api-access-rfrxp" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045994    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f20bcca-4ad5-4241-bf25-c59211b8eb9c" volumeName="kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-tls" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046006    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046018    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3acab04e-53c1-4ceb-b571-246d23c55934" volumeName="kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-etc-tuned" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046032    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22ec490-f623-4cda-a693-e6669a1a3324" volumeName="kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-config" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046052    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa3f9f6e-77c4-4053-9289-65469d56d6ac" volumeName="kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-daemon-config" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.045985    5420 factory.go:55] Registering systemd factory
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046066    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ccec32-8bf2-4f47-8d30-abd99b513d1b" volumeName="kubernetes.io/configmap/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-serviceca" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046071    5420 factory.go:223] Registration of the systemd container factory successfully
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046077    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f20bcca-4ad5-4241-bf25-c59211b8eb9c" volumeName="kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046089    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b3d0acf-5ab2-47ce-99ef-21f5565fbd99" volumeName="kubernetes.io/projected/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-kube-api-access-x7zln" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046100    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1705c77-7f46-4719-8dd1-51993028ab1e" volumeName="kubernetes.io/projected/e1705c77-7f46-4719-8dd1-51993028ab1e-kube-api-access-jh2mh" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046112    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" volumeName="kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 13:09:19.046146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046126    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3acab04e-53c1-4ceb-b571-246d23c55934" volumeName="kubernetes.io/projected/3acab04e-53c1-4ceb-b571-246d23c55934-kube-api-access-j48g5" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046174    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67dddf6c-5f0c-4894-aeb2-c26135c32c07" volumeName="kubernetes.io/secret/67dddf6c-5f0c-4894-aeb2-c26135c32c07-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046194    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22ec490-f623-4cda-a693-e6669a1a3324" volumeName="kubernetes.io/secret/a22ec490-f623-4cda-a693-e6669a1a3324-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046224    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f20bcca-4ad5-4241-bf25-c59211b8eb9c" volumeName="kubernetes.io/projected/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-kube-api-access-st6rh" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046241    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22aec86e-8e32-4e9b-82d7-218a2ceb3926" volumeName="kubernetes.io/secret/22aec86e-8e32-4e9b-82d7-218a2ceb3926-agent-certs" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046254    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046267    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046281    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8db0e985-d36d-485d-91ae-917f751a61a1" volumeName="kubernetes.io/projected/8db0e985-d36d-485d-91ae-917f751a61a1-kube-api-access-jvvx7" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046293    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22ec490-f623-4cda-a693-e6669a1a3324" volumeName="kubernetes.io/projected/a22ec490-f623-4cda-a693-e6669a1a3324-kube-api-access-t9bt2" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046306    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa3f9f6e-77c4-4053-9289-65469d56d6ac" volumeName="kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cni-binary-copy" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046320    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c40982af-b5a4-4f58-b35e-93208bfd793e" volumeName="kubernetes.io/projected/c40982af-b5a4-4f58-b35e-93208bfd793e-kube-api-access-dw4ml" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046338    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046359    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ffdebd2-3a79-4eb9-baf4-87b91d78ac01" volumeName="kubernetes.io/projected/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-kube-api-access-46pvl" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046375    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b3d0acf-5ab2-47ce-99ef-21f5565fbd99" volumeName="kubernetes.io/secret/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046392    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67dddf6c-5f0c-4894-aeb2-c26135c32c07" volumeName="kubernetes.io/configmap/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046411    5420 factory.go:153] Registering CRI-O factory
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046429    5420 factory.go:223] Registration of the crio container factory successfully
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046447    5420 factory.go:103] Registering Raw factory
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046461    5420 manager.go:1199] Started watching for new ooms in manager
Nov 03 13:09:19.047059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046786    5420 manager.go:319] Starting recovery of all containers
Nov 03 13:09:19.048523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.048500    5420 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:09:19.049254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.046411    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67dddf6c-5f0c-4894-aeb2-c26135c32c07" volumeName="kubernetes.io/empty-dir/67dddf6c-5f0c-4894-aeb2-c26135c32c07-data-volume" seLinuxMountContext=""
Nov 03 13:09:19.049327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049272    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" volumeName="kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 13:09:19.049327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049299    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 13:09:19.049327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049313    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 13:09:19.049327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049326    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" volumeName="kubernetes.io/projected/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-kube-api-access-d94tq" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049338    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22ec490-f623-4cda-a693-e6669a1a3324" volumeName="kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049351    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d2aa382-9410-48f2-b139-3d9d12a26b3b" volumeName="kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049365    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049376    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f20bcca-4ad5-4241-bf25-c59211b8eb9c" volumeName="kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-metrics-client-ca" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049388    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049400    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ffdebd2-3a79-4eb9-baf4-87b91d78ac01" volumeName="kubernetes.io/empty-dir/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-tmp-dir" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049412    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ccec32-8bf2-4f47-8d30-abd99b513d1b" volumeName="kubernetes.io/projected/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-kube-api-access-xfr92" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049425    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8db0e985-d36d-485d-91ae-917f751a61a1" volumeName="kubernetes.io/configmap/8db0e985-d36d-485d-91ae-917f751a61a1-iptables-alerter-script" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049439    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22ec490-f623-4cda-a693-e6669a1a3324" volumeName="kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-env-overrides" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049460    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049488    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049500    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049511    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 13:09:19.049524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049526    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" volumeName="kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-binary-copy" seLinuxMountContext=""
Nov 03 13:09:19.050174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049544    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df" volumeName="kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret" seLinuxMountContext=""
Nov 03 13:09:19.050174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049556    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="181d5ad2-c22d-4091-b14b-38cd5c105e3f" volumeName="kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 13:09:19.050174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049567    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3acab04e-53c1-4ceb-b571-246d23c55934" volumeName="kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-tmp" seLinuxMountContext=""
Nov 03 13:09:19.050174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049581    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1705c77-7f46-4719-8dd1-51993028ab1e" volumeName="kubernetes.io/configmap/e1705c77-7f46-4719-8dd1-51993028ab1e-config-volume" seLinuxMountContext=""
Nov 03 13:09:19.050174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049593    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1705c77-7f46-4719-8dd1-51993028ab1e" volumeName="kubernetes.io/empty-dir/e1705c77-7f46-4719-8dd1-51993028ab1e-tmp-dir" seLinuxMountContext=""
Nov 03 13:09:19.050174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049604    5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1705c77-7f46-4719-8dd1-51993028ab1e" volumeName="kubernetes.io/secret/e1705c77-7f46-4719-8dd1-51993028ab1e-metrics-tls" seLinuxMountContext=""
Nov 03 13:09:19.050174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049615    5420 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 13:09:19.050174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.049622    5420 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 13:09:19.055849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.055817    5420 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 13:09:19.065776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.065755    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:09:19.068038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.068021    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:09:19.074889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.074856    5420 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 13:09:19.076124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.076098    5420 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 13:09:19.076206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.076125    5420 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 13:09:19.076206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.076150    5420 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 13:09:19.076206 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:19.076194    5420 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 13:09:19.080913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.080627    5420 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 13:09:19.088932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.088906    5420 generic.go:356] "Generic (PLEG): container finished" podID="0f20bcca-4ad5-4241-bf25-c59211b8eb9c" containerID="694028e85bd36a494c34429b23be4f9647d646b2e747f5335251007ed1efe4dd" exitCode=0
Nov 03 13:09:19.092534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.092510    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="446ef6194a788dc976dd1d623bec2f8ce8bf712cafd38b7587a6665f18b2dcc9" exitCode=0
Nov 03 13:09:19.092534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.092530    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="0ff963af78644ae46cffe1209cc5993ab2c4890093e0d2d7569976c702985cb6" exitCode=0
Nov 03 13:09:19.092689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.092538    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="bcb3044779e1819ac303837ae0c471ffa896e64362a01f29533c5d5ca3e4b975" exitCode=0
Nov 03 13:09:19.092689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.092549    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="be55a4e2642c3f9745e252ae26d30a978510c0fd68c207054473608456dcfbde" exitCode=0
Nov 03 13:09:19.092689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.092554    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="c0de7afca2a06488f054cd538633d621fe6978db272f76e3b0a0b17648e34c65" exitCode=0
Nov 03 13:09:19.092689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.092559    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642" containerID="1071962ef8ae0c6c7074bda3434258583b1ff624ce9e957c57ba500aaf41b332" exitCode=0
Nov 03 13:09:19.100056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.100031    5420 generic.go:356] "Generic (PLEG): container finished" podID="be41ec8017368ca412baa7a38539e26d" containerID="804f151da7bc7c982099c6b8a5c5ea470afea576edb7e9440be29c2fbad5761f" exitCode=0
Nov 03 13:09:19.109167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.109115    5420 generic.go:356] "Generic (PLEG): container finished" podID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397" exitCode=0
Nov 03 13:09:19.119091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.119064    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:09:19.119196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.119105    5420 generic.go:356] "Generic (PLEG): container finished" podID="aa3f9f6e-77c4-4053-9289-65469d56d6ac" containerID="07937cc9f49ed441f408d67a0b68238b645f3d1ec5151e3bb47208c7976a69e9" exitCode=1
Nov 03 13:09:19.136359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.136337    5420 manager.go:324] Recovery completed
Nov 03 13:09:19.156075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156048    5420 cpu_manager.go:226] "Starting" policy="none"
Nov 03 13:09:19.156075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156065    5420 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 13:09:19.156075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156082    5420 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 13:09:19.156290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156203    5420 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 13:09:19.156290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156229    5420 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 13:09:19.156290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156248    5420 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 13:09:19.156290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156253    5420 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 13:09:19.156290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156258    5420 policy_none.go:50] "Start"
Nov 03 13:09:19.156290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156267    5420 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 13:09:19.156290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156276    5420 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 13:09:19.156475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156400    5420 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 13:09:19.156475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156407    5420 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 13:09:19.156475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.156413    5420 policy_none.go:44] "Start"
Nov 03 13:09:19.161205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.160774    5420 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 13:09:19.166334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.166317    5420 manager.go:341] "Starting Device Plugin manager"
Nov 03 13:09:19.166434 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:19.166342    5420 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 13:09:19.166434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.166352    5420 server.go:87] "Starting device plugin registration server"
Nov 03 13:09:19.166613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.166598    5420 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 13:09:19.166678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.166612    5420 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 13:09:19.166730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.166704    5420 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 13:09:19.166938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.166816    5420 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 13:09:19.166938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.166916    5420 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 13:09:19.166938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.166924    5420 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 13:09:19.167116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.166959    5420 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:09:19.166904172Z","UUID":"26ecfec9-237b-43b2-bfcd-28d78ee21a02","Handler":null,"Name":"","Endpoint":""}
Nov 03 13:09:19.168366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.168347    5420 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 13:09:19.168464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.168381    5420 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 13:09:19.168609 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:19.168589    5420 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 13:09:19.176347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.176255    5420 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"]
Nov 03 13:09:19.179776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.179728    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"1bebcb79842815c8633e47e5468f5a5a","Type":"ContainerStarted","Data":"5ba0985e3f25fc441569710e76078af955eaacd76e8b65fbc159f7733ebd259c"}
Nov 03 13:09:19.179874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.179785    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"1bebcb79842815c8633e47e5468f5a5a","Type":"ContainerStarted","Data":"5dc2b06e68791904bccd9bd8df8e3adf716a720cd98d047704d55b78877e53e3"}
Nov 03 13:09:19.179874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.179803    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"be41ec8017368ca412baa7a38539e26d","Type":"ContainerStarted","Data":"bb6f5730b3c12611b316a0830442bf9add9a2bdc68d9be5ee987710388838d6f"}
Nov 03 13:09:19.179874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.179817    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"be41ec8017368ca412baa7a38539e26d","Type":"ContainerDied","Data":"804f151da7bc7c982099c6b8a5c5ea470afea576edb7e9440be29c2fbad5761f"}
Nov 03 13:09:19.179874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.179843    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal" event={"ID":"be41ec8017368ca412baa7a38539e26d","Type":"ContainerStarted","Data":"24900b96e5aa02946a235af3656ad6fe62d6cad2008ca8569568f6034cd5d130"}
Nov 03 13:09:19.179874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.179855    5420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.180115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.180001    5420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.191425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.191405    5420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:09:19.191498 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:19.191451    5420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.191724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.191710    5420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:09:19.191756 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:19.191747    5420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.256263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.256203    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.256401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.256279    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.256401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.256325    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bebcb79842815c8633e47e5468f5a5a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"1bebcb79842815c8633e47e5468f5a5a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.267324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.267305    5420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 13:09:19.269898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.269883    5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 13:09:19.269990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.269913    5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 13:09:19.269990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.269924    5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-248.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 13:09:19.269990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.269963    5420 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.281990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.281972    5420 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.282055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.282040    5420 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.357088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.357054    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bebcb79842815c8633e47e5468f5a5a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"1bebcb79842815c8633e47e5468f5a5a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.357293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.357098    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.357293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.357120    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.357293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.357066    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bebcb79842815c8633e47e5468f5a5a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"1bebcb79842815c8633e47e5468f5a5a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.357293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.357157    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:19.357293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:19.357192    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be41ec8017368ca412baa7a38539e26d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" (UID: \"be41ec8017368ca412baa7a38539e26d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:20.018499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.018461    5420 apiserver.go:52] "Watching apiserver"
Nov 03 13:09:20.033747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.033721    5420 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 13:09:20.034267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.034243    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-lr88l","openshift-multus/network-metrics-daemon-5gj6w","openshift-ovn-kubernetes/ovnkube-node-k4pjp","kube-system/konnectivity-agent-729bm","openshift-insights/insights-runtime-extractor-l8wlg","kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal","openshift-image-registry/node-ca-xptcl","openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t","openshift-multus/multus-additional-cni-plugins-k445k","openshift-network-operator/iptables-alerter-zkrzz","openshift-dns/dns-default-qdgff","openshift-network-diagnostics/network-check-target-dnqj9","openshift-dns/node-resolver-lx28x","openshift-monitoring/prometheus-k8s-0","openshift-cluster-node-tuning-operator/tuned-72hxg","openshift-ingress-canary/ingress-canary-zfk5q","openshift-multus/multus-4bn8t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal","openshift-monitoring/node-exporter-ggskq"]
Nov 03 13:09:20.041046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.041015    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 13:09:20.041046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.041039    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.041241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.041055    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-knm89\""
Nov 03 13:09:20.041241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.041062    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.041241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.041108    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.042432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042409    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 13:09:20.042525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042483    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.042525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042513    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.042609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042521    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-psqgf\""
Nov 03 13:09:20.042609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042527    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.042609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042535    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 13:09:20.042807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042784    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 13:09:20.042807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042825    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 13:09:20.042977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042872    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 13:09:20.042977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.042898    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.044294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044272    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 13:09:20.044381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044293    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 13:09:20.044381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044306    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s4cl2\""
Nov 03 13:09:20.044381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044319    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.044381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044380    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vwt8x\""
Nov 03 13:09:20.044656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044631    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-62cxl\""
Nov 03 13:09:20.044704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044671    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hltl5\""
Nov 03 13:09:20.044788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044721    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 13:09:20.044788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044725    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mrd6\""
Nov 03 13:09:20.044788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044738    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.044788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044768    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.044943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.044800    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jqlk5\""
Nov 03 13:09:20.046057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046036    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wv2nv\""
Nov 03 13:09:20.046057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046052    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 13:09:20.046162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046061    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 13:09:20.046162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046095    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.046162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046097    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.046386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046368    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 13:09:20.046541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046526    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-569wf\""
Nov 03 13:09:20.046607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046553    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.046607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046572    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.046607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046573    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 13:09:20.046607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046593    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.046607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046599    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 13:09:20.046782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.046574    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.048022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.047982    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.048099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.048030    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 13:09:20.048173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.048099    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d95tp\""
Nov 03 13:09:20.048173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.048112    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c596b\""
Nov 03 13:09:20.048339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.048192    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 13:09:20.048438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.048423    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjbx2\""
Nov 03 13:09:20.048500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.048450    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 13:09:20.048500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.048456    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.048500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.048461    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 13:09:20.048605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.048529    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 13:09:20.049762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.049748    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 13:09:20.049829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.049764    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wd8kt\""
Nov 03 13:09:20.049829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.049776    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.049829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.049780    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.049829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.049817    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 13:09:20.050224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.050196    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mfl7n\""
Nov 03 13:09:20.050274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.050252    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gkggt\""
Nov 03 13:09:20.050316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.050251    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 13:09:20.050316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.050290    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 13:09:20.051843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.051832    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 13:09:20.052109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.052093    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 13:09:20.052175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.052107    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 13:09:20.052175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.052119    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 13:09:20.052175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.052109    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 13:09:20.053921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.053907    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 13:09:20.053975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.053908    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 13:09:20.055174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.055155    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s2gms\""
Nov 03 13:09:20.055174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.055165    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 13:09:20.055289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.055188    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 13:09:20.055747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.055731    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 13:09:20.055878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.055748    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-85dfg\""
Nov 03 13:09:20.055878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.055794    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 13:09:20.057532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.057517    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 13:09:20.057620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.057566    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 13:09:20.057620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.057611    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 13:09:20.057710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.057621    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 13:09:20.057710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.057629    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-uhmup5ug6m5s\""
Nov 03 13:09:20.057710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.057625    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 13:09:20.057710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.057653    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 13:09:20.061356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061334    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:20.061444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061379    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.061444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061412    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:20.061544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061439    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-k8s-cni-cncf-io\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.061544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061479    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-conf-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.061544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061522    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 13:09:20.061544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061514    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-etc-kubernetes\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.061701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061579    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.061701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061637    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-slash\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.061786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061766    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-etc-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.061832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061806    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-metrics-client-ca\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.061881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061830    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.061881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061852    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-original-pull-secret\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:20.061881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061867    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.061881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061874    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2mh\" (UniqueName: \"kubernetes.io/projected/e1705c77-7f46-4719-8dd1-51993028ab1e-kube-api-access-jh2mh\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:20.062062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061914    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.062062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.061977    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-registration-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.062062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062017    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4ml\" (UniqueName: \"kubernetes.io/projected/c40982af-b5a4-4f58-b35e-93208bfd793e-kube-api-access-dw4ml\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:20.062062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062026    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.062198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062064    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:20.062198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062075    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-metrics-client-ca\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.062198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062171    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-dbus\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:20.062348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062202    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.062348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062246    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67dddf6c-5f0c-4894-aeb2-c26135c32c07-data-volume\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.062348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062267    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-log-socket\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.062348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062281    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.062348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062303    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.062348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062325    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67dddf6c-5f0c-4894-aeb2-c26135c32c07-data-volume\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.062348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062334    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-system-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062367    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062392    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062422    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062424    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062480    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6x6m\" (UniqueName: \"kubernetes.io/projected/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-api-access-m6x6m\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062487    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062507    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-etc-tuned\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062515    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062534    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cnibin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062561    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfr92\" (UniqueName: \"kubernetes.io/projected/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-kube-api-access-xfr92\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062565    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-etc-tuned\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062585    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-daemon-config\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062612    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-multus-certs\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062588    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062638    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-host\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:09:20.062664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062663    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-var-lib-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062689    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062730    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-daemon-config\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062744    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8db0e985-d36d-485d-91ae-917f751a61a1-iptables-alerter-script\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062796    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22aec86e-8e32-4e9b-82d7-218a2ceb3926-konnectivity-ca\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062822    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-wtmp\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062839    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfm9p\" (UniqueName: \"kubernetes.io/projected/aa3f9f6e-77c4-4053-9289-65469d56d6ac-kube-api-access-zfm9p\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062854    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062855    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062870    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-node-log\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062896    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bt2\" (UniqueName: \"kubernetes.io/projected/a22ec490-f623-4cda-a693-e6669a1a3324-kube-api-access-t9bt2\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062910    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8db0e985-d36d-485d-91ae-917f751a61a1-iptables-alerter-script\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062925    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062938    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-conf\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062973    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22aec86e-8e32-4e9b-82d7-218a2ceb3926-konnectivity-ca\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.062980    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-host\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063048    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063101    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063119    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063134    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c40982af-b5a4-4f58-b35e-93208bfd793e-cert\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063148    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-ovn\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063164    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-binary-copy\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063179    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063194    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-root\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063303    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-tmp\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.063324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063337    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c40982af-b5a4-4f58-b35e-93208bfd793e-cert\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063375    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063420    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3acab04e-53c1-4ceb-b571-246d23c55934-tmp\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063480    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063507    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-tmp-dir\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063528    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063543    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-textfile\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063556    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063564    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-socket-dir-parent\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063594    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-binary-copy\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063631    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063635    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-tmp-dir\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063665    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-textfile\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063690    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-bin\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063706    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cnibin\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063708    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063732    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063757    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a22ec490-f623-4cda-a693-e6669a1a3324-ovn-node-metrics-cert\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063769    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063810    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-kubernetes\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063822    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063855    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-sys\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063886    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-lib-modules\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063921    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-bin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.063977    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064006    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d2aa382-9410-48f2-b139-3d9d12a26b3b-metrics-certs\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064011    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a22ec490-f623-4cda-a693-e6669a1a3324-ovn-node-metrics-cert\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064039    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvx7\" (UniqueName: \"kubernetes.io/projected/8db0e985-d36d-485d-91ae-917f751a61a1-kube-api-access-jvvx7\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064056    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-kubelet\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064071    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-device-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064097    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064125    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pvl\" (UniqueName: \"kubernetes.io/projected/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-kube-api-access-46pvl\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064189    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-netns\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.064244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064250    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-etc-selinux\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064279    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064297    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67dddf6c-5f0c-4894-aeb2-c26135c32c07-crio-socket\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064321    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-system-cni-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064351    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064352    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064379    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-kubelet\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064385    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064479    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-config\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064498    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064500    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1705c77-7f46-4719-8dd1-51993028ab1e-metrics-tls\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064526    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064548    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1705c77-7f46-4719-8dd1-51993028ab1e-tmp-dir\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064576    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-var-lib-kubelet\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064600    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-serviceca\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064618    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-config\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064620    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1705c77-7f46-4719-8dd1-51993028ab1e-tmp-dir\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064652    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1705c77-7f46-4719-8dd1-51993028ab1e-metrics-tls\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064697    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064718    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064734    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-systemd-units\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064751    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrxp\" (UniqueName: \"kubernetes.io/projected/0d2aa382-9410-48f2-b139-3d9d12a26b3b-kube-api-access-rfrxp\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064765    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-run\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064854    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-kubelet-config\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064896    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.064933    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7xv\" (UniqueName: \"kubernetes.io/projected/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kube-api-access-4g7xv\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065051    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065147    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-systemd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065145    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-serviceca\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065261    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zln\" (UniqueName: \"kubernetes.io/projected/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-kube-api-access-x7zln\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065305    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065396    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065431    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065487    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065531    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8db0e985-d36d-485d-91ae-917f751a61a1-host-slash\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065548    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-netns\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065562    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065571    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-systemd\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065586    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-multus\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065628    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67dddf6c-5f0c-4894-aeb2-c26135c32c07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065682    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48g5\" (UniqueName: \"kubernetes.io/projected/3acab04e-53c1-4ceb-b571-246d23c55934-kube-api-access-j48g5\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.065709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065708    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065756    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-tls\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065849    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-modprobe-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065870    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67dddf6c-5f0c-4894-aeb2-c26135c32c07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065919    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065947    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-hostroot\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.065976    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysconfig\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066007    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-tls\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066019    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-socket-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066054    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-hosts-file\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066082    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94tq\" (UniqueName: \"kubernetes.io/projected/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-kube-api-access-d94tq\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066105    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cni-binary-copy\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066206    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1705c77-7f46-4719-8dd1-51993028ab1e-config-volume\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066300    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-sys-fs\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066332    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-script-lib\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066347    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1705c77-7f46-4719-8dd1-51993028ab1e-config-volume\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066354    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cni-binary-copy\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066370    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22aec86e-8e32-4e9b-82d7-218a2ceb3926-agent-certs\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066451    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6rh\" (UniqueName: \"kubernetes.io/projected/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-kube-api-access-st6rh\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066468    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066496    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-ovnkube-script-lib\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066503    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066522    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-netd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066538    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-env-overrides\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066552    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-sys\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066570    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5cj\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066594    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066711    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066864    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a22ec490-f623-4cda-a693-e6669a1a3324-env-overrides\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066878    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22aec86e-8e32-4e9b-82d7-218a2ceb3926-agent-certs\") pod \"konnectivity-agent-729bm\" (UID: \"22aec86e-8e32-4e9b-82d7-218a2ceb3926\") " pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:09:20.067205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.066940    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.068333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.067458    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.068514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.068495    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2mh\" (UniqueName: \"kubernetes.io/projected/e1705c77-7f46-4719-8dd1-51993028ab1e-kube-api-access-jh2mh\") pod \"dns-default-qdgff\" (UID: \"e1705c77-7f46-4719-8dd1-51993028ab1e\") " pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:20.068592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.068574    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4ml\" (UniqueName: \"kubernetes.io/projected/c40982af-b5a4-4f58-b35e-93208bfd793e-kube-api-access-dw4ml\") pod \"ingress-canary-zfk5q\" (UID: \"c40982af-b5a4-4f58-b35e-93208bfd793e\") " pod="openshift-ingress-canary/ingress-canary-zfk5q"
Nov 03 13:09:20.068736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.068717    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4nl\" (UniqueName: \"kubernetes.io/projected/09bd042b-b415-46cc-a7af-aa4a57ed11d5-kube-api-access-ww4nl\") pod \"network-check-target-dnqj9\" (UID: \"09bd042b-b415-46cc-a7af-aa4a57ed11d5\") " pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:20.068788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.068771    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6x6m\" (UniqueName: \"kubernetes.io/projected/67dddf6c-5f0c-4894-aeb2-c26135c32c07-kube-api-access-m6x6m\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.069269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.069254    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfm9p\" (UniqueName: \"kubernetes.io/projected/aa3f9f6e-77c4-4053-9289-65469d56d6ac-kube-api-access-zfm9p\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.069595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.069577    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfr92\" (UniqueName: \"kubernetes.io/projected/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-kube-api-access-xfr92\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:09:20.069907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.069889    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bt2\" (UniqueName: \"kubernetes.io/projected/a22ec490-f623-4cda-a693-e6669a1a3324-kube-api-access-t9bt2\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.070914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.070894    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pvl\" (UniqueName: \"kubernetes.io/projected/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-kube-api-access-46pvl\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:09:20.071022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.071007    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvx7\" (UniqueName: \"kubernetes.io/projected/8db0e985-d36d-485d-91ae-917f751a61a1-kube-api-access-jvvx7\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:09:20.071353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.071335    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrxp\" (UniqueName: \"kubernetes.io/projected/0d2aa382-9410-48f2-b139-3d9d12a26b3b-kube-api-access-rfrxp\") pod \"network-metrics-daemon-5gj6w\" (UID: \"0d2aa382-9410-48f2-b139-3d9d12a26b3b\") " pod="openshift-multus/network-metrics-daemon-5gj6w"
Nov 03 13:09:20.071692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.071671    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zln\" (UniqueName: \"kubernetes.io/projected/5b3d0acf-5ab2-47ce-99ef-21f5565fbd99-kube-api-access-x7zln\") pod \"monitoring-plugin-77f4f76594-xnfmz\" (UID: \"5b3d0acf-5ab2-47ce-99ef-21f5565fbd99\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:20.071757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.071744    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7xv\" (UniqueName: \"kubernetes.io/projected/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kube-api-access-4g7xv\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.076057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.076034    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48g5\" (UniqueName: \"kubernetes.io/projected/3acab04e-53c1-4ceb-b571-246d23c55934-kube-api-access-j48g5\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.076245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.076225    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94tq\" (UniqueName: \"kubernetes.io/projected/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-kube-api-access-d94tq\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.076287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.076230    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6rh\" (UniqueName: \"kubernetes.io/projected/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-kube-api-access-st6rh\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.076994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.076979    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5cj\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj\") pod \"prometheus-k8s-0\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:20.128853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.128829    5420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:20.129016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.128943    5420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:20.140462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.140437    5420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:09:20.140610 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:20.140477    5420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:20.141324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.141305    5420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 13:09:20.141440 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:20.141340    5420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.us-east-2.compute.internal"
Nov 03 13:09:20.144556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.144541    5420 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 13:09:20.167149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167123    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-log-socket\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.167149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167152    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167172    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-system-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167188    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cnibin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167207    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-multus-certs\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167243    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-host\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167274    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-var-lib-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167270    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167292    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-wtmp\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167292    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-multus-certs\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167315    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-node-log\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167297    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6ccec32-8bf2-4f47-8d30-abd99b513d1b-host\") pod \"node-ca-xptcl\" (UID: \"c6ccec32-8bf2-4f47-8d30-abd99b513d1b\") " pod="openshift-image-registry/node-ca-xptcl"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167274    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-system-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167332    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-conf\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167342    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-node-log\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167358    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-host\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.167369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167376    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-ovn\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167279    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-cnibin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167397    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167317    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-log-socket\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167409    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-ovn\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167408    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-node-exporter-wtmp\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167429    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-root\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167448    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-host\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167455    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-conf\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167462    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-socket-dir-parent\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167311    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-var-lib-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167490    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-bin\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167500    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167523    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-bin\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167526    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cnibin\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167503    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-root\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167547    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-socket-dir-parent\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167553    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-kubernetes\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167578    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-sys\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167593    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-lib-modules\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167607    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-bin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167623    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-kubernetes\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167641    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-sys\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167641    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-bin\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167560    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-cnibin\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167663    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-kubelet\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167692    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-device-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167717    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-lib-modules\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167720    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-netns\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167734    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-kubelet\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167751    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-etc-selinux\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167755    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-netns\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167762    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-device-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167771    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67dddf6c-5f0c-4894-aeb2-c26135c32c07-crio-socket\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167786    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-system-cni-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167801    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-kubelet\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167837    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642-system-cni-dir\") pod \"multus-additional-cni-plugins-k445k\" (UID: \"2c1e9fa4-b3c4-4f21-a4cb-9d846a39f642\") " pod="openshift-multus/multus-additional-cni-plugins-k445k"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167840    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67dddf6c-5f0c-4894-aeb2-c26135c32c07-crio-socket\") pod \"insights-runtime-extractor-l8wlg\" (UID: \"67dddf6c-5f0c-4894-aeb2-c26135c32c07\") " pod="openshift-insights/insights-runtime-extractor-l8wlg"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167862    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-kubelet\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167875    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-etc-selinux\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.168011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167967    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-var-lib-kubelet\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.167824    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-var-lib-kubelet\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168280    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168298    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-systemd-units\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168311    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-run\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168325    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-kubelet-config\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168339    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168360    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-systemd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168387    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8db0e985-d36d-485d-91ae-917f751a61a1-host-slash\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168395    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-kubelet-config\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168362    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-systemd-units\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168409    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-netns\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168405    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168422    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-run\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168455    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-systemd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168468    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-run-netns\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168460    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysctl-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168496    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-systemd\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168505    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8db0e985-d36d-485d-91ae-917f751a61a1-host-slash\") pod \"iptables-alerter-zkrzz\" (UID: \"8db0e985-d36d-485d-91ae-917f751a61a1\") " pod="openshift-network-operator/iptables-alerter-zkrzz"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168523    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-multus\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168549    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-modprobe-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168575    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-var-lib-cni-multus\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168580    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-systemd\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168573    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168601    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-run-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168612    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-hostroot\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168636    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysconfig\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168657    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-hostroot\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168661    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-socket-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168680    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-modprobe-d\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168691    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-hosts-file\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168710    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-socket-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168721    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-sys-fs\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168724    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3acab04e-53c1-4ceb-b571-246d23c55934-etc-sysconfig\") pod \"tuned-72hxg\" (UID: \"3acab04e-53c1-4ceb-b571-246d23c55934\") " pod="openshift-cluster-node-tuning-operator/tuned-72hxg"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168749    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-netd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168766    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-sys\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168785    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168796    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-cni-netd\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168764    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ffdebd2-3a79-4eb9-baf4-87b91d78ac01-hosts-file\") pod \"node-resolver-lx28x\" (UID: \"1ffdebd2-3a79-4eb9-baf4-87b91d78ac01\") " pod="openshift-dns/node-resolver-lx28x"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168806    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-k8s-cni-cncf-io\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168827    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168832    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-conf-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168843    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f20bcca-4ad5-4241-bf25-c59211b8eb9c-sys\") pod \"node-exporter-ggskq\" (UID: \"0f20bcca-4ad5-4241-bf25-c59211b8eb9c\") " pod="openshift-monitoring/node-exporter-ggskq"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168849    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-host-run-k8s-cni-cncf-io\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168800    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-sys-fs\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168862    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-etc-kubernetes\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168870    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-conf-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168886    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-slash\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168900    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-etc-kubernetes\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168901    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-etc-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168923    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-etc-openvswitch\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168926    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168949    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-registration-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168950    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a22ec490-f623-4cda-a693-e6669a1a3324-host-slash\") pod \"ovnkube-node-k4pjp\" (UID: \"a22ec490-f623-4cda-a693-e6669a1a3324\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168969    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-dbus\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168983    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dca216a1-acaf-4fe6-b833-4fc7e25ad427-registration-dir\") pod \"aws-ebs-csi-driver-node-ws88t\" (UID: \"dca216a1-acaf-4fe6-b833-4fc7e25ad427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ws88t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.168999    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa3f9f6e-77c4-4053-9289-65469d56d6ac-multus-cni-dir\") pod \"multus-4bn8t\" (UID: \"aa3f9f6e-77c4-4053-9289-65469d56d6ac\") " pod="openshift-multus/multus-4bn8t"
Nov 03 13:09:20.169435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.169014    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df-dbus\") pod \"global-pull-secret-syncer-lr88l\" (UID: \"2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df\") " pod="kube-system/global-pull-secret-syncer-lr88l"
Nov 03 13:09:20.343164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:20.343143    5420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:09:21.205577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:21.205532    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:21.328078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:21.328024    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:22.505694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:22.505662    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qdgff"
Nov 03 13:09:23.440033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:23.439939    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:23.459940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:23.459913    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:24.145538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:24.145504    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dnqj9" event={"ID":"09bd042b-b415-46cc-a7af-aa4a57ed11d5","Type":"ContainerStarted","Data":"e96d10d9744144512277fec892acbb48f1282a8348ac872403357b55e1aa8252"}
Nov 03 13:09:24.145963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:24.145941    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:24.729270 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:24.729233    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:09:26.282584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:26.282553    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xnfmz"
Nov 03 13:09:27.651752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:27.651723    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:27.951917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:27.951839    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:28.134424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:28.134386    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:28.157750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:28.157727    5420 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 13:09:28.259943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:28.259915    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:29.657320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:29.657282    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:29.813120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:29.813081    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:30.568844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:30.568817    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4pjp"
Nov 03 13:09:36.146915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:36.146882    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-729bm"
Nov 03 13:09:44.656221 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:44.656187    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:44.815985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:44.815941    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:49.253706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:49.253670    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:49.429579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:49.429549    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:54.297722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.297681    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"]
Nov 03 13:09:54.341070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.341039    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"]
Nov 03 13:09:54.341271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.341170    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.345300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.345243    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 13:09:54.345300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.345259    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 13:09:54.345300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.345294    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 13:09:54.345538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.345519    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 13:09:54.345594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.345553    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 13:09:54.345594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.345571    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-h98np\""
Nov 03 13:09:54.351119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.351092    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 13:09:54.403251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.403192    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-federate-client-tls\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.403251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.403248    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.403439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.403293    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bjm\" (UniqueName: \"kubernetes.io/projected/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-kube-api-access-82bjm\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.403439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.403411    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-metrics-client-ca\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.403521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.403447    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-serving-certs-ca-bundle\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.403521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.403473    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.403592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.403520    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-telemeter-client-tls\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.403592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.403568    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-secret-telemeter-client\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.504660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.504614    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-serving-certs-ca-bundle\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.504852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.504665    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.504852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.504738    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-telemeter-client-tls\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.504852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.504775    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-secret-telemeter-client\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.504852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.504828    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-federate-client-tls\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.505072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.504855    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.505072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.504919    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bjm\" (UniqueName: \"kubernetes.io/projected/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-kube-api-access-82bjm\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.505072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.504974    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-metrics-client-ca\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.505247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.505158    5420 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 13:09:54.505616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.505592    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-serving-certs-ca-bundle\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.505734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.505692    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-metrics-client-ca\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.505794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.505779    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.507618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.507591    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-telemeter-client-tls\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.507715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.507593    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.507916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.507895    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-secret-telemeter-client\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.507957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.507925    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-federate-client-tls\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.517914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.517890    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bjm\" (UniqueName: \"kubernetes.io/projected/bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57-kube-api-access-82bjm\") pod \"telemeter-client-5df6484b9c-zjh5n\" (UID: \"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57\") " pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.613170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.613080    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 13:09:54.613626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.613577    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="prometheus" containerID="cri-o://e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5" gracePeriod=600
Nov 03 13:09:54.613783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.613606    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy" containerID="cri-o://d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f" gracePeriod=600
Nov 03 13:09:54.613783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.613623    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="thanos-sidecar" containerID="cri-o://56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4" gracePeriod=600
Nov 03 13:09:54.613783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.613649    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy-web" containerID="cri-o://47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0" gracePeriod=600
Nov 03 13:09:54.613783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.613628    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9" gracePeriod=600
Nov 03 13:09:54.613783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.613659    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="config-reloader" containerID="cri-o://e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67" gracePeriod=600
Nov 03 13:09:54.692901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.692873    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"
Nov 03 13:09:54.736870 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:54.735821    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:09:54.875338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.875314    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:54.996676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:54.996645    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5df6484b9c-zjh5n"]
Nov 03 13:09:54.999568 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:54.999536    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbe3d46_7f9a_4fb5_a2c3_779c138c4a57.slice/crio-3cba55b3db25b5c42c2e611d10a5cc6d1aa255fd4c651e8380ee71ee99bb6034 WatchSource:0}: Error finding container 3cba55b3db25b5c42c2e611d10a5cc6d1aa255fd4c651e8380ee71ee99bb6034: Status 404 returned error can't find the container with id 3cba55b3db25b5c42c2e611d10a5cc6d1aa255fd4c651e8380ee71ee99bb6034
Nov 03 13:09:55.008252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008229    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008265    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008285    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008306    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008328    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008364    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008394    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008421    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008457    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008496    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008526    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008560    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008595    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008642    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008675    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008679    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:55.008972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008693    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:55.008972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008718    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008749    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008789    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008824    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.008972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.008863    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle\") pod \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\" (UID: \"181d5ad2-c22d-4091-b14b-38cd5c105e3f\") "
Nov 03 13:09:55.009451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.009081    5420 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-metrics-client-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.009451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.009099    5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.009451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.009116    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:55.009451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.009232    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:55.009699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.009473    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:55.010195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.009932    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:55.011683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.011658    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:55.012163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.012138    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:55.012562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.012502    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:55.012659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.012612    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "kube-api-access-7r5cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:55.013066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.013028    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:55.013066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.013043    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:55.013245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.013062    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:55.013314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.013245    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:55.013372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.013305    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:55.013605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.013584    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:55.013681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.013643    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:09:55.014233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.014192    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:55.014633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.014617    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:55.021799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.021777    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config" pod "181d5ad2-c22d-4091-b14b-38cd5c105e3f" (UID: "181d5ad2-c22d-4091-b14b-38cd5c105e3f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:55.110308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110279    5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110304    5420 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110316    5420 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110326    5420 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110335    5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-db\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110344    5420 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110353    5420 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110362    5420 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-tls-assets\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110371    5420 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/181d5ad2-c22d-4091-b14b-38cd5c105e3f-config-out\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110380    5420 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-metrics-client-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110388    5420 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110397    5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110405    5420 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-grpc-tls\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110413    5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/181d5ad2-c22d-4091-b14b-38cd5c105e3f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110422    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5cj\" (UniqueName: \"kubernetes.io/projected/181d5ad2-c22d-4091-b14b-38cd5c105e3f-kube-api-access-7r5cj\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110451    5420 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-web-config\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110461    5420 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-kube-rbac-proxy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.110475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.110469    5420 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/181d5ad2-c22d-4091-b14b-38cd5c105e3f-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:55.151224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.151136    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dnqj9"
Nov 03 13:09:55.247032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.246999    5420 generic.go:356] "Generic (PLEG): container finished" podID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerID="567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9" exitCode=0
Nov 03 13:09:55.247032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247023    5420 generic.go:356] "Generic (PLEG): container finished" podID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerID="d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f" exitCode=0
Nov 03 13:09:55.247032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247029    5420 generic.go:356] "Generic (PLEG): container finished" podID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerID="47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0" exitCode=0
Nov 03 13:09:55.247032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247035    5420 generic.go:356] "Generic (PLEG): container finished" podID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerID="56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4" exitCode=0
Nov 03 13:09:55.247032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247041    5420 generic.go:356] "Generic (PLEG): container finished" podID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerID="e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67" exitCode=0
Nov 03 13:09:55.247032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247046    5420 generic.go:356] "Generic (PLEG): container finished" podID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerID="e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5" exitCode=0
Nov 03 13:09:55.247485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247070    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerDied","Data":"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"}
Nov 03 13:09:55.247485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247118    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.247485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247157    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerDied","Data":"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"}
Nov 03 13:09:55.247485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247201    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerDied","Data":"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"}
Nov 03 13:09:55.247485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247252    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerDied","Data":"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"}
Nov 03 13:09:55.247485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247258    5420 scope.go:122] "RemoveContainer" containerID="567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"
Nov 03 13:09:55.247485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247293    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerDied","Data":"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"}
Nov 03 13:09:55.247485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247332    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerDied","Data":"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"}
Nov 03 13:09:55.247485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.247373    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"181d5ad2-c22d-4091-b14b-38cd5c105e3f","Type":"ContainerDied","Data":"542253c26f759b90a14fae7ce45623859641b7cb81629b2477d96a28adee6d08"}
Nov 03 13:09:55.248441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.248376    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n" event={"ID":"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57","Type":"ContainerStarted","Data":"3cba55b3db25b5c42c2e611d10a5cc6d1aa255fd4c651e8380ee71ee99bb6034"}
Nov 03 13:09:55.254734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.254716    5420 scope.go:122] "RemoveContainer" containerID="d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"
Nov 03 13:09:55.261024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.261008    5420 scope.go:122] "RemoveContainer" containerID="47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"
Nov 03 13:09:55.267613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.267595    5420 scope.go:122] "RemoveContainer" containerID="56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"
Nov 03 13:09:55.274528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.274505    5420 scope.go:122] "RemoveContainer" containerID="e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"
Nov 03 13:09:55.281145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.281130    5420 scope.go:122] "RemoveContainer" containerID="e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"
Nov 03 13:09:55.287803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.287788    5420 scope.go:122] "RemoveContainer" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"
Nov 03 13:09:55.290963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.290930    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 13:09:55.294940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.294919    5420 scope.go:122] "RemoveContainer" containerID="567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"
Nov 03 13:09:55.295206 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:55.295179    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": container with ID starting with 567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9 not found: ID does not exist" containerID="567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"
Nov 03 13:09:55.295300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.295241    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"} err="failed to get container status \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": rpc error: code = NotFound desc = could not find container \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": container with ID starting with 567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9 not found: ID does not exist"
Nov 03 13:09:55.295300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.295277    5420 scope.go:122] "RemoveContainer" containerID="d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"
Nov 03 13:09:55.295542 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:55.295524    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": container with ID starting with d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f not found: ID does not exist" containerID="d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"
Nov 03 13:09:55.295581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.295550    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"} err="failed to get container status \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": rpc error: code = NotFound desc = could not find container \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": container with ID starting with d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f not found: ID does not exist"
Nov 03 13:09:55.295581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.295565    5420 scope.go:122] "RemoveContainer" containerID="47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"
Nov 03 13:09:55.295793 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:55.295776    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": container with ID starting with 47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0 not found: ID does not exist" containerID="47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"
Nov 03 13:09:55.295832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.295799    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"} err="failed to get container status \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": rpc error: code = NotFound desc = could not find container \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": container with ID starting with 47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0 not found: ID does not exist"
Nov 03 13:09:55.295832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.295812    5420 scope.go:122] "RemoveContainer" containerID="56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"
Nov 03 13:09:55.296053 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:55.296032    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": container with ID starting with 56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4 not found: ID does not exist" containerID="56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"
Nov 03 13:09:55.296124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.296061    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"} err="failed to get container status \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": rpc error: code = NotFound desc = could not find container \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": container with ID starting with 56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4 not found: ID does not exist"
Nov 03 13:09:55.296124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.296082    5420 scope.go:122] "RemoveContainer" containerID="e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"
Nov 03 13:09:55.296331 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:55.296311    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": container with ID starting with e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67 not found: ID does not exist" containerID="e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"
Nov 03 13:09:55.296374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.296361    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"} err="failed to get container status \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": rpc error: code = NotFound desc = could not find container \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": container with ID starting with e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67 not found: ID does not exist"
Nov 03 13:09:55.296413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.296374    5420 scope.go:122] "RemoveContainer" containerID="e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"
Nov 03 13:09:55.296637 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:55.296621    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": container with ID starting with e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5 not found: ID does not exist" containerID="e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"
Nov 03 13:09:55.296683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.296641    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"} err="failed to get container status \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": rpc error: code = NotFound desc = could not find container \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": container with ID starting with e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5 not found: ID does not exist"
Nov 03 13:09:55.296683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.296655    5420 scope.go:122] "RemoveContainer" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"
Nov 03 13:09:55.296869 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:09:55.296853    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": container with ID starting with d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397 not found: ID does not exist" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"
Nov 03 13:09:55.296911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.296884    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"} err="failed to get container status \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": rpc error: code = NotFound desc = could not find container \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": container with ID starting with d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397 not found: ID does not exist"
Nov 03 13:09:55.296911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.296899    5420 scope.go:122] "RemoveContainer" containerID="567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"
Nov 03 13:09:55.297168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.297147    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"} err="failed to get container status \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": rpc error: code = NotFound desc = could not find container \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": container with ID starting with 567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9 not found: ID does not exist"
Nov 03 13:09:55.297264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.297168    5420 scope.go:122] "RemoveContainer" containerID="d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"
Nov 03 13:09:55.297427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.297407    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"} err="failed to get container status \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": rpc error: code = NotFound desc = could not find container \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": container with ID starting with d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f not found: ID does not exist"
Nov 03 13:09:55.297491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.297428    5420 scope.go:122] "RemoveContainer" containerID="47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"
Nov 03 13:09:55.297818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.297790    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"} err="failed to get container status \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": rpc error: code = NotFound desc = could not find container \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": container with ID starting with 47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0 not found: ID does not exist"
Nov 03 13:09:55.297818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.297818    5420 scope.go:122] "RemoveContainer" containerID="56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"
Nov 03 13:09:55.298350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.298330    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"} err="failed to get container status \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": rpc error: code = NotFound desc = could not find container \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": container with ID starting with 56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4 not found: ID does not exist"
Nov 03 13:09:55.298350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.298350    5420 scope.go:122] "RemoveContainer" containerID="e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"
Nov 03 13:09:55.298607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.298588    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"} err="failed to get container status \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": rpc error: code = NotFound desc = could not find container \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": container with ID starting with e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67 not found: ID does not exist"
Nov 03 13:09:55.298695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.298616    5420 scope.go:122] "RemoveContainer" containerID="e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"
Nov 03 13:09:55.298864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.298843    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"} err="failed to get container status \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": rpc error: code = NotFound desc = could not find container \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": container with ID starting with e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5 not found: ID does not exist"
Nov 03 13:09:55.298952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.298866    5420 scope.go:122] "RemoveContainer" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"
Nov 03 13:09:55.299128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.299107    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"} err="failed to get container status \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": rpc error: code = NotFound desc = could not find container \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": container with ID starting with d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397 not found: ID does not exist"
Nov 03 13:09:55.299128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.299126    5420 scope.go:122] "RemoveContainer" containerID="567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"
Nov 03 13:09:55.299474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.299449    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"} err="failed to get container status \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": rpc error: code = NotFound desc = could not find container \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": container with ID starting with 567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9 not found: ID does not exist"
Nov 03 13:09:55.299550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.299475    5420 scope.go:122] "RemoveContainer" containerID="d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"
Nov 03 13:09:55.299721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.299702    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 13:09:55.299771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.299707    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"} err="failed to get container status \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": rpc error: code = NotFound desc = could not find container \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": container with ID starting with d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f not found: ID does not exist"
Nov 03 13:09:55.299771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.299745    5420 scope.go:122] "RemoveContainer" containerID="47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"
Nov 03 13:09:55.299945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.299928    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"} err="failed to get container status \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": rpc error: code = NotFound desc = could not find container \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": container with ID starting with 47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0 not found: ID does not exist"
Nov 03 13:09:55.299997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.299946    5420 scope.go:122] "RemoveContainer" containerID="56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"
Nov 03 13:09:55.300158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300141    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"} err="failed to get container status \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": rpc error: code = NotFound desc = could not find container \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": container with ID starting with 56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4 not found: ID does not exist"
Nov 03 13:09:55.300200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300159    5420 scope.go:122] "RemoveContainer" containerID="e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"
Nov 03 13:09:55.300374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300356    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"} err="failed to get container status \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": rpc error: code = NotFound desc = could not find container \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": container with ID starting with e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67 not found: ID does not exist"
Nov 03 13:09:55.300420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300375    5420 scope.go:122] "RemoveContainer" containerID="e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"
Nov 03 13:09:55.300569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300554    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"} err="failed to get container status \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": rpc error: code = NotFound desc = could not find container \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": container with ID starting with e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5 not found: ID does not exist"
Nov 03 13:09:55.300607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300569    5420 scope.go:122] "RemoveContainer" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"
Nov 03 13:09:55.300751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300736    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"} err="failed to get container status \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": rpc error: code = NotFound desc = could not find container \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": container with ID starting with d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397 not found: ID does not exist"
Nov 03 13:09:55.300795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300752    5420 scope.go:122] "RemoveContainer" containerID="567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"
Nov 03 13:09:55.300976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300960    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"} err="failed to get container status \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": rpc error: code = NotFound desc = could not find container \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": container with ID starting with 567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9 not found: ID does not exist"
Nov 03 13:09:55.301021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.300977    5420 scope.go:122] "RemoveContainer" containerID="d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"
Nov 03 13:09:55.301162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301144    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"} err="failed to get container status \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": rpc error: code = NotFound desc = could not find container \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": container with ID starting with d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f not found: ID does not exist"
Nov 03 13:09:55.301204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301163    5420 scope.go:122] "RemoveContainer" containerID="47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"
Nov 03 13:09:55.301384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301368    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"} err="failed to get container status \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": rpc error: code = NotFound desc = could not find container \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": container with ID starting with 47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0 not found: ID does not exist"
Nov 03 13:09:55.301426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301385    5420 scope.go:122] "RemoveContainer" containerID="56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"
Nov 03 13:09:55.301585 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301569    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"} err="failed to get container status \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": rpc error: code = NotFound desc = could not find container \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": container with ID starting with 56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4 not found: ID does not exist"
Nov 03 13:09:55.301623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301585    5420 scope.go:122] "RemoveContainer" containerID="e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"
Nov 03 13:09:55.301774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301757    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"} err="failed to get container status \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": rpc error: code = NotFound desc = could not find container \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": container with ID starting with e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67 not found: ID does not exist"
Nov 03 13:09:55.301821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301775    5420 scope.go:122] "RemoveContainer" containerID="e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"
Nov 03 13:09:55.301952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301936    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"} err="failed to get container status \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": rpc error: code = NotFound desc = could not find container \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": container with ID starting with e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5 not found: ID does not exist"
Nov 03 13:09:55.301992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.301951    5420 scope.go:122] "RemoveContainer" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"
Nov 03 13:09:55.302142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302127    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"} err="failed to get container status \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": rpc error: code = NotFound desc = could not find container \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": container with ID starting with d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397 not found: ID does not exist"
Nov 03 13:09:55.302193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302141    5420 scope.go:122] "RemoveContainer" containerID="567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"
Nov 03 13:09:55.302358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302340    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"} err="failed to get container status \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": rpc error: code = NotFound desc = could not find container \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": container with ID starting with 567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9 not found: ID does not exist"
Nov 03 13:09:55.302401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302359    5420 scope.go:122] "RemoveContainer" containerID="d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"
Nov 03 13:09:55.302543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302526    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"} err="failed to get container status \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": rpc error: code = NotFound desc = could not find container \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": container with ID starting with d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f not found: ID does not exist"
Nov 03 13:09:55.302617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302544    5420 scope.go:122] "RemoveContainer" containerID="47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"
Nov 03 13:09:55.302722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302706    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"} err="failed to get container status \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": rpc error: code = NotFound desc = could not find container \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": container with ID starting with 47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0 not found: ID does not exist"
Nov 03 13:09:55.302766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302722    5420 scope.go:122] "RemoveContainer" containerID="56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"
Nov 03 13:09:55.302901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302886    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"} err="failed to get container status \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": rpc error: code = NotFound desc = could not find container \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": container with ID starting with 56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4 not found: ID does not exist"
Nov 03 13:09:55.302954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.302901    5420 scope.go:122] "RemoveContainer" containerID="e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"
Nov 03 13:09:55.303101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303087    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"} err="failed to get container status \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": rpc error: code = NotFound desc = could not find container \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": container with ID starting with e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67 not found: ID does not exist"
Nov 03 13:09:55.303142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303102    5420 scope.go:122] "RemoveContainer" containerID="e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"
Nov 03 13:09:55.303293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303278    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"} err="failed to get container status \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": rpc error: code = NotFound desc = could not find container \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": container with ID starting with e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5 not found: ID does not exist"
Nov 03 13:09:55.303336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303293    5420 scope.go:122] "RemoveContainer" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"
Nov 03 13:09:55.303494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303477    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"} err="failed to get container status \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": rpc error: code = NotFound desc = could not find container \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": container with ID starting with d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397 not found: ID does not exist"
Nov 03 13:09:55.303537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303494    5420 scope.go:122] "RemoveContainer" containerID="567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"
Nov 03 13:09:55.303678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303662    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9"} err="failed to get container status \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": rpc error: code = NotFound desc = could not find container \"567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9\": container with ID starting with 567af2554e4668702f15c8a228833232aa26f6e3590f9088911cb550a3db23d9 not found: ID does not exist"
Nov 03 13:09:55.303719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303677    5420 scope.go:122] "RemoveContainer" containerID="d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"
Nov 03 13:09:55.303865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303848    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f"} err="failed to get container status \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": rpc error: code = NotFound desc = could not find container \"d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f\": container with ID starting with d07cf5d426d8dbe7398a9dc6afa09139cf20ca989e21df211bee9e86215b316f not found: ID does not exist"
Nov 03 13:09:55.303907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.303865    5420 scope.go:122] "RemoveContainer" containerID="47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"
Nov 03 13:09:55.304059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.304042    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0"} err="failed to get container status \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": rpc error: code = NotFound desc = could not find container \"47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0\": container with ID starting with 47126bbc2535e53649766af09385717fe97411cecca9fc73af7ae4eb8f78bfe0 not found: ID does not exist"
Nov 03 13:09:55.304104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.304060    5420 scope.go:122] "RemoveContainer" containerID="56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"
Nov 03 13:09:55.304277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.304260    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4"} err="failed to get container status \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": rpc error: code = NotFound desc = could not find container \"56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4\": container with ID starting with 56037325b9633192ad18ae3c8770e0a26e180e933af61005058651a0702021e4 not found: ID does not exist"
Nov 03 13:09:55.304324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.304277    5420 scope.go:122] "RemoveContainer" containerID="e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"
Nov 03 13:09:55.304481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.304448    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67"} err="failed to get container status \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": rpc error: code = NotFound desc = could not find container \"e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67\": container with ID starting with e2214fe132ce31a5c1513b4cc13c2bad6226cf71b931d2f121eab86ddfd1dd67 not found: ID does not exist"
Nov 03 13:09:55.304519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.304482    5420 scope.go:122] "RemoveContainer" containerID="e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"
Nov 03 13:09:55.304675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.304657    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5"} err="failed to get container status \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": rpc error: code = NotFound desc = could not find container \"e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5\": container with ID starting with e39bd47d98086701809079cfe188f6a935d030171b1680b4f2a5e95214b0aff5 not found: ID does not exist"
Nov 03 13:09:55.304720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.304675    5420 scope.go:122] "RemoveContainer" containerID="d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"
Nov 03 13:09:55.304865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.304849    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397"} err="failed to get container status \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": rpc error: code = NotFound desc = could not find container \"d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397\": container with ID starting with d857a5dd47c155e29d798dfe7b06d3fcf5d777fc0009125e0a6a4ca0f4ae7397 not found: ID does not exist"
Nov 03 13:09:55.333367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333333    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 13:09:55.333626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333610    5420 cpu_manager.go:414] "containerMap: removing container" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy"
Nov 03 13:09:55.333697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333631    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy"
Nov 03 13:09:55.333697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333653    5420 cpu_manager.go:414] "containerMap: removing container" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="thanos-sidecar"
Nov 03 13:09:55.333697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333661    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="thanos-sidecar"
Nov 03 13:09:55.333697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333675    5420 cpu_manager.go:414] "containerMap: removing container" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy-web"
Nov 03 13:09:55.333697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333683    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy-web"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333700    5420 cpu_manager.go:414] "containerMap: removing container" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="prometheus"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333741    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="prometheus"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333763    5420 cpu_manager.go:414] "containerMap: removing container" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy-thanos"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333772    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy-thanos"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333789    5420 cpu_manager.go:414] "containerMap: removing container" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="config-reloader"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333797    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="config-reloader"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333807    5420 cpu_manager.go:414] "containerMap: removing container" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="init-config-reloader"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333814    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="init-config-reloader"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333875    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="prometheus"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333888    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy-thanos"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333897    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="config-reloader"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333907    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="init-config-reloader"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333916    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333925    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="thanos-sidecar"
Nov 03 13:09:55.333945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.333934    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" containerName="kube-rbac-proxy-web"
Nov 03 13:09:55.337553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.337532    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.342236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.342202    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 13:09:55.342336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.342270    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-85dfg\""
Nov 03 13:09:55.342522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.342507    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 13:09:55.343486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.343468    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 13:09:55.343486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.343486    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 13:09:55.343642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.343499    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 13:09:55.343642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.343504    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 13:09:55.343642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.343504    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 13:09:55.343642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.343613    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 13:09:55.345112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.345095    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 13:09:55.345247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.345233    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 13:09:55.345376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.345363    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-uhmup5ug6m5s\""
Nov 03 13:09:55.345419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.345384    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 13:09:55.347081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.347062    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 13:09:55.352303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.352286    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 13:09:55.352439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.352414    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 13:09:55.352622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.352605    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 13:09:55.357790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.357771    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 13:09:55.411914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.411835    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.411914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.411867    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.411914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.411885    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55439d7f-aaca-4102-b1b3-07f4bf6669f1-config-out\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.411914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.411900    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412030    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghrw\" (UniqueName: \"kubernetes.io/projected/55439d7f-aaca-4102-b1b3-07f4bf6669f1-kube-api-access-kghrw\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412062    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412079    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412098    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412186    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412277    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-config\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412294    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412325    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412340    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412397    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412450    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412522    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412558    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55439d7f-aaca-4102-b1b3-07f4bf6669f1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412584    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-web-config\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412611    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.412795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.412656    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.513728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.513658    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.513728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.513690    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.513728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.513715    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.513986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.513739    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.513986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.513866    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.513986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.513902    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55439d7f-aaca-4102-b1b3-07f4bf6669f1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.513986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.513926    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-web-config\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.513986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.513951    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.513988    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514022    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514046    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514068    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55439d7f-aaca-4102-b1b3-07f4bf6669f1-config-out\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514089    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514141    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghrw\" (UniqueName: \"kubernetes.io/projected/55439d7f-aaca-4102-b1b3-07f4bf6669f1-kube-api-access-kghrw\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514173    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514198    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514245    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514278    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514470    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-config\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514555    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.514911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.514647    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.515432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.515343    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.517188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.516613    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.517188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.516855    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.517188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.517079    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.517797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.517773    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.517969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.517949    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.518178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.518152    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/55439d7f-aaca-4102-b1b3-07f4bf6669f1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.518920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.518882    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.519446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.519421    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.520809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.520789    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.521380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.521362    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55439d7f-aaca-4102-b1b3-07f4bf6669f1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.521484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.521457    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-config\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.522634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.522611    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55439d7f-aaca-4102-b1b3-07f4bf6669f1-config-out\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.522759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.522738    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.522825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.522770    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-web-config\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.523039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.523021    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.524765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.524740    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.524969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.524828    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/55439d7f-aaca-4102-b1b3-07f4bf6669f1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.527415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.527393    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghrw\" (UniqueName: \"kubernetes.io/projected/55439d7f-aaca-4102-b1b3-07f4bf6669f1-kube-api-access-kghrw\") pod \"prometheus-k8s-0\" (UID: \"55439d7f-aaca-4102-b1b3-07f4bf6669f1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.654086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.654045    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:09:55.949719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:55.949626    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 13:09:55.953489 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:09:55.953458    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55439d7f_aaca_4102_b1b3_07f4bf6669f1.slice/crio-3671aebfdfd53be2526c966e8dd7b5bb22a9c8dd25ee9bd6f77f66a303387725 WatchSource:0}: Error finding container 3671aebfdfd53be2526c966e8dd7b5bb22a9c8dd25ee9bd6f77f66a303387725: Status 404 returned error can't find the container with id 3671aebfdfd53be2526c966e8dd7b5bb22a9c8dd25ee9bd6f77f66a303387725
Nov 03 13:09:56.254926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:56.254893    5420 generic.go:356] "Generic (PLEG): container finished" podID="55439d7f-aaca-4102-b1b3-07f4bf6669f1" containerID="3011b452df69d3d7dbbe3b0ce71584f4f7a684ec5e967a60779a73fbddaf77e1" exitCode=0
Nov 03 13:09:56.255246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:56.255199    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55439d7f-aaca-4102-b1b3-07f4bf6669f1","Type":"ContainerDied","Data":"3011b452df69d3d7dbbe3b0ce71584f4f7a684ec5e967a60779a73fbddaf77e1"}
Nov 03 13:09:56.255335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:56.255309    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55439d7f-aaca-4102-b1b3-07f4bf6669f1","Type":"ContainerStarted","Data":"3671aebfdfd53be2526c966e8dd7b5bb22a9c8dd25ee9bd6f77f66a303387725"}
Nov 03 13:09:57.082002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.081920    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181d5ad2-c22d-4091-b14b-38cd5c105e3f" path="/var/lib/kubelet/pods/181d5ad2-c22d-4091-b14b-38cd5c105e3f/volumes"
Nov 03 13:09:57.260849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.260807    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n" event={"ID":"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57","Type":"ContainerStarted","Data":"f65516861cd79a72a6f7c5c140b7840ed370ba101b4a433046c45bc5ac52d83f"}
Nov 03 13:09:57.261020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.260876    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n" event={"ID":"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57","Type":"ContainerStarted","Data":"ea9e649d114a6114b9a3eecb7b32d84278c5ef63b48a4f85caf234f7eed1334e"}
Nov 03 13:09:57.261020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.260903    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n" event={"ID":"bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57","Type":"ContainerStarted","Data":"2bd4ea4e983322d26116b2a13aeb581a8ec704eeb321835434ccfab5c2321115"}
Nov 03 13:09:57.267621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.267585    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55439d7f-aaca-4102-b1b3-07f4bf6669f1","Type":"ContainerStarted","Data":"0cfaabbad2bf8f842f8133763ad6314eee0bd6b77929833c3e1d8b162d7c6f34"}
Nov 03 13:09:57.267835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.267667    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55439d7f-aaca-4102-b1b3-07f4bf6669f1","Type":"ContainerStarted","Data":"b79ee419b5e86422edaad8fdbcd476bb21d735c1e5dcf8075fb343a53bdc2465"}
Nov 03 13:09:57.267835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.267694    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55439d7f-aaca-4102-b1b3-07f4bf6669f1","Type":"ContainerStarted","Data":"29cb02ae33067c372d1e2b20833a191d6c3cde24da15acf48938ea90b7609a68"}
Nov 03 13:09:57.267835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.267721    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55439d7f-aaca-4102-b1b3-07f4bf6669f1","Type":"ContainerStarted","Data":"b424de664b0ad56f5935b2f16d3cd3018795b10ae26b121aae2955b44775ff16"}
Nov 03 13:09:57.267835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.267744    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55439d7f-aaca-4102-b1b3-07f4bf6669f1","Type":"ContainerStarted","Data":"ec868121cee86d8aba4d7137b167cb5914d538d4c8bf466b4ba66fddd49e3aeb"}
Nov 03 13:09:57.267835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.267765    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55439d7f-aaca-4102-b1b3-07f4bf6669f1","Type":"ContainerStarted","Data":"e7858bb2d01e8af02d6b2d94bdaec9bb1079e29b7d224195d0e16a4626425ce3"}
Nov 03 13:09:57.287867 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.287800    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5df6484b9c-zjh5n" podStartSLOduration=1.617007697 podStartE2EDuration="3.287787456s" totalImagesPullingTime="1.670779759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:57.287197838 +0000 UTC m=+38.364962887" watchObservedRunningTime="2026-11-03 13:09:57.287787456 +0000 UTC m=+38.365552475"
Nov 03 13:09:57.321825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.321775    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.302990652 podStartE2EDuration="2.321760115s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.769463ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:57.319770842 +0000 UTC m=+38.397535867" watchObservedRunningTime="2026-11-03 13:09:57.321760115 +0000 UTC m=+38.399525142"
Nov 03 13:09:57.544581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.544549    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:57.710580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:57.710537    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:58.150742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:58.150708    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:58.331493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:09:58.331456    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:00.655143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:00.655089    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:10:16.368592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:16.368558    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:22.369792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:22.369757    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:22.504043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:22.503995    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:24.729786 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:10:24.729753    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:10:33.146969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:33.146928    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:10:33.161947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:33.161923    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:10:33.399720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:33.399645    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 13:10:34.576261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:34.576203    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:34.721104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:34.721068    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:41.212044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:41.212015    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:41.342022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:41.341990    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:41.558047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:41.557951    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:41.685548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:41.685512    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:45.676493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:45.676462    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:45.809661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:45.809626    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:47.204055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:47.204008    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:47.331156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:47.331125    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:54.729511 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:10:54.729474    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:10:56.741164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:56.741083    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:59.633516 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:59.633482    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:59.766789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:10:59.766754    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:04.301831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:11:04.301797    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:04.436676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:11:04.436642    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:16.422965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:11:16.422928    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:24.729009 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:11:24.728964    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:11:29.842987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:11:29.842937    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:54.729779 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:11:54.729740    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:12:16.462819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:12:16.462779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:24.729255 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:12:24.729191    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:12:30.449431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:12:30.449353    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:54.729114 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:12:54.729073    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:12:59.476328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:12:59.476294    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:00.383730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.383697    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"]
Nov 03 13:13:00.386782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.386760    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.392738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.392718    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 13:13:00.392947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.392929    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 13:13:00.393029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.392931    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-f9lrs\""
Nov 03 13:13:00.411186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.411163    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"]
Nov 03 13:13:00.557377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.557328    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.557377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.557383    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdnt\" (UniqueName: \"kubernetes.io/projected/6808393b-7c55-4452-b277-089ad22d7ae9-kube-api-access-5pdnt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.557802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.557482    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.658612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.658525    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.658612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.658581    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdnt\" (UniqueName: \"kubernetes.io/projected/6808393b-7c55-4452-b277-089ad22d7ae9-kube-api-access-5pdnt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.658829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.658651    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.658928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.658904    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.659043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.659022    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.668655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.668628    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdnt\" (UniqueName: \"kubernetes.io/projected/6808393b-7c55-4452-b277-089ad22d7ae9-kube-api-access-5pdnt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.701892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.701863    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:00.982557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:00.982535    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"]
Nov 03 13:13:00.985117 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:00.985085    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6808393b_7c55_4452_b277_089ad22d7ae9.slice/crio-f32c2f2b5e5073f9744d36ca59e53e60a9b9c4d3ba74ecc33a797bfab27343e9 WatchSource:0}: Error finding container f32c2f2b5e5073f9744d36ca59e53e60a9b9c4d3ba74ecc33a797bfab27343e9: Status 404 returned error can't find the container with id f32c2f2b5e5073f9744d36ca59e53e60a9b9c4d3ba74ecc33a797bfab27343e9
Nov 03 13:13:01.227201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.227162    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:01.596535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.595390    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"]
Nov 03 13:13:01.598715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.598691    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.607324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.607299    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"]
Nov 03 13:13:01.611464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.611411    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9wg28\""
Nov 03 13:13:01.688242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.688200    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"]
Nov 03 13:13:01.691415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.691390    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"]
Nov 03 13:13:01.691554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.691525    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.694866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.694799    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.699892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.699867    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"]
Nov 03 13:13:01.703007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.702936    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"]
Nov 03 13:13:01.704590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.704566    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-wpmk5\""
Nov 03 13:13:01.704699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.704689    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-gc28m\""
Nov 03 13:13:01.768628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.768595    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.768817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.768655    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qsnf\" (UniqueName: \"kubernetes.io/projected/f7036dbc-3508-410c-8213-41c039b28a7f-kube-api-access-7qsnf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.768817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.768772    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.852400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.852323    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4" event={"ID":"6808393b-7c55-4452-b277-089ad22d7ae9","Type":"ContainerStarted","Data":"f32c2f2b5e5073f9744d36ca59e53e60a9b9c4d3ba74ecc33a797bfab27343e9"}
Nov 03 13:13:01.869715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.869684    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.869887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.869722    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbz8x\" (UniqueName: \"kubernetes.io/projected/dd04039e-43be-4b17-87d4-326e0f1ac053-kube-api-access-mbz8x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.869887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.869746    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qsnf\" (UniqueName: \"kubernetes.io/projected/f7036dbc-3508-410c-8213-41c039b28a7f-kube-api-access-7qsnf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.869887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.869823    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.869887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.869871    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.870093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.869891    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.870093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.869912    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.870093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.869975    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9g6q\" (UniqueName: \"kubernetes.io/projected/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-kube-api-access-z9g6q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.870093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.870064    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.870286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.870264    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.870411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.870371    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.880052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.880020    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qsnf\" (UniqueName: \"kubernetes.io/projected/f7036dbc-3508-410c-8213-41c039b28a7f-kube-api-access-7qsnf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.921882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.921849    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:01.971243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.971191    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.971416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.971274    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g6q\" (UniqueName: \"kubernetes.io/projected/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-kube-api-access-z9g6q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.971416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.971350    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.971416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.971382    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbz8x\" (UniqueName: \"kubernetes.io/projected/dd04039e-43be-4b17-87d4-326e0f1ac053-kube-api-access-mbz8x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.971416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.971409    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.971623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.971447    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.971680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.971630    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.971856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.971836    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.971949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.971924    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.972092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.972075    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.982700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.980894    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9g6q\" (UniqueName: \"kubernetes.io/projected/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-kube-api-access-z9g6q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:01.983476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.983457    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbz8x\" (UniqueName: \"kubernetes.io/projected/dd04039e-43be-4b17-87d4-326e0f1ac053-kube-api-access-mbz8x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:01.995470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:01.995444    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:02.013760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:02.013738    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:02.020018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:02.019996    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:02.200640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:02.200571    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"]
Nov 03 13:13:02.203817 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:02.203787    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7036dbc_3508_410c_8213_41c039b28a7f.slice/crio-638513b46998094f2bd8b9920cb406d5019161312509fce463315cd9c783d2b2 WatchSource:0}: Error finding container 638513b46998094f2bd8b9920cb406d5019161312509fce463315cd9c783d2b2: Status 404 returned error can't find the container with id 638513b46998094f2bd8b9920cb406d5019161312509fce463315cd9c783d2b2
Nov 03 13:13:02.292566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:02.292536    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"]
Nov 03 13:13:02.305222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:02.305185    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"]
Nov 03 13:13:02.309021 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:02.308998    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f77b6e_8c5a_4c8e_9860_f1391bd970fe.slice/crio-a35b3e36a7c10961f0654f0e996fb2dda181dd3e2c3fd880b9290aa98bd65394 WatchSource:0}: Error finding container a35b3e36a7c10961f0654f0e996fb2dda181dd3e2c3fd880b9290aa98bd65394: Status 404 returned error can't find the container with id a35b3e36a7c10961f0654f0e996fb2dda181dd3e2c3fd880b9290aa98bd65394
Nov 03 13:13:02.858591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:02.858521    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92" event={"ID":"dd04039e-43be-4b17-87d4-326e0f1ac053","Type":"ContainerStarted","Data":"dd310548d077231bb8aff3e32bea32a36252bcecc125644a7c7ef3ba9f8a5082"}
Nov 03 13:13:02.861361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:02.861304    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d" event={"ID":"f7036dbc-3508-410c-8213-41c039b28a7f","Type":"ContainerStarted","Data":"638513b46998094f2bd8b9920cb406d5019161312509fce463315cd9c783d2b2"}
Nov 03 13:13:02.862746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:02.862722    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc" event={"ID":"02f77b6e-8c5a-4c8e-9860-f1391bd970fe","Type":"ContainerStarted","Data":"a35b3e36a7c10961f0654f0e996fb2dda181dd3e2c3fd880b9290aa98bd65394"}
Nov 03 13:13:05.170746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:05.170714    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:05.297058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:05.297010    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:06.022243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.020593    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"]
Nov 03 13:13:06.026872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.026555    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.034594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.034571    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"]
Nov 03 13:13:06.212866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.212819    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntqsr\" (UniqueName: \"kubernetes.io/projected/e7006cab-6027-4ad2-9673-58474e10d530-kube-api-access-ntqsr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.213307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.212896    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.213307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.212957    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.314665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.314036    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.314665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.314097    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqsr\" (UniqueName: \"kubernetes.io/projected/e7006cab-6027-4ad2-9673-58474e10d530-kube-api-access-ntqsr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.314665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.314131    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.314665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.314532    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.314665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.314567    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.331487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.331450    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqsr\" (UniqueName: \"kubernetes.io/projected/e7006cab-6027-4ad2-9673-58474e10d530-kube-api-access-ntqsr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.353923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.353862    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:06.642693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.642620    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"]
Nov 03 13:13:06.649094 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:06.649059    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7006cab_6027_4ad2_9673_58474e10d530.slice/crio-945f45b5e9411d09b907a49e94b58886d26d8e9349212b530713755a814bd61b WatchSource:0}: Error finding container 945f45b5e9411d09b907a49e94b58886d26d8e9349212b530713755a814bd61b: Status 404 returned error can't find the container with id 945f45b5e9411d09b907a49e94b58886d26d8e9349212b530713755a814bd61b
Nov 03 13:13:06.883581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:06.883529    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs" event={"ID":"e7006cab-6027-4ad2-9673-58474e10d530","Type":"ContainerStarted","Data":"945f45b5e9411d09b907a49e94b58886d26d8e9349212b530713755a814bd61b"}
Nov 03 13:13:08.062850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.062789    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:08.189306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.189221    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:08.639482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.638023    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"]
Nov 03 13:13:08.643711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.642986    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.655199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.653704    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 13:13:08.655199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.653897    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 13:13:08.655199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.653993    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 13:13:08.655199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.654396    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"]
Nov 03 13:13:08.685770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.685715    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nxhjw\""
Nov 03 13:13:08.725449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.725418    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"]
Nov 03 13:13:08.731267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.731235    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.744321    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a0c725-e696-46db-89fa-e022d39904c4-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.744342    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"]
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.744374    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.745257    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.745294    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a0c725-e696-46db-89fa-e022d39904c4-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.745401    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a0c725-e696-46db-89fa-e022d39904c4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.745452    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.745494    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.745524    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8968f39e-7bbd-4d21-afa6-67116c521656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.745713    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.745757    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.745793    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.747288    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.747323    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.747992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.747345    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8968f39e-7bbd-4d21-afa6-67116c521656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.747992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.747367    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4hd\" (UniqueName: \"kubernetes.io/projected/26a0c725-e696-46db-89fa-e022d39904c4-kube-api-access-xc4hd\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.747992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.747385    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.747992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.747400    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.747992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.747417    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbg75\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-kube-api-access-sbg75\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.749115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.749095    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-p4p9q\""
Nov 03 13:13:08.766827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.766793    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"]
Nov 03 13:13:08.771580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.771539    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:08.778475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.778447    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"]
Nov 03 13:13:08.778675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.778655    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.848827    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.848883    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.848918    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.848944    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.848970    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.848993    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8968f39e-7bbd-4d21-afa6-67116c521656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849023    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4hd\" (UniqueName: \"kubernetes.io/projected/26a0c725-e696-46db-89fa-e022d39904c4-kube-api-access-xc4hd\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849052    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849075    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849102    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbg75\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-kube-api-access-sbg75\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849145    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849183    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849250    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a0c725-e696-46db-89fa-e022d39904c4-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849280    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849310    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jvvr\" (UniqueName: \"kubernetes.io/projected/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-kube-api-access-4jvvr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849381    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849407    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a0c725-e696-46db-89fa-e022d39904c4-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849442    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a0c725-e696-46db-89fa-e022d39904c4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849480    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849500    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.849517    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8968f39e-7bbd-4d21-afa6-67116c521656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.850322    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8968f39e-7bbd-4d21-afa6-67116c521656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.851267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.850949    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.852603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.851603    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.852603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.851676    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.852603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.851823    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.852603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.852016    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a0c725-e696-46db-89fa-e022d39904c4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.852603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.852133    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.852603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.852155    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.852603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.852387    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.852603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.852571    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.874444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.874368    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4hd\" (UniqueName: \"kubernetes.io/projected/26a0c725-e696-46db-89fa-e022d39904c4-kube-api-access-xc4hd\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.874771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.874743    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.874919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.874894    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.875189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.875167    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8968f39e-7bbd-4d21-afa6-67116c521656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.875448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.875423    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a0c725-e696-46db-89fa-e022d39904c4-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.878757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.878484    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbg75\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-kube-api-access-sbg75\") pod \"maas-default-gateway-openshift-default-84587756dc-6sg42\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.878757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.878550    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a0c725-e696-46db-89fa-e022d39904c4-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.878757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.878714    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a0c725-e696-46db-89fa-e022d39904c4-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn\" (UID: \"26a0c725-e696-46db-89fa-e022d39904c4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:08.950095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.950016    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:08.950095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.950066    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:08.950360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.950114    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jvvr\" (UniqueName: \"kubernetes.io/projected/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-kube-api-access-4jvvr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:08.950620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.950592    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:08.950830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.950783    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:08.951325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.951265    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:08.962755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.962729    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:08.963615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:08.963532    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jvvr\" (UniqueName: \"kubernetes.io/projected/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-kube-api-access-4jvvr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:09.057109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.056857    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:09.081919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.081814    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:09.093454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.093327    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:09.288455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.288409    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"]
Nov 03 13:13:09.293103 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:09.293074    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8968f39e_7bbd_4d21_afa6_67116c521656.slice/crio-f260255f8b880f748e9e877a3787aef8a83b896a85c6f2188740b24dfe37c963 WatchSource:0}: Error finding container f260255f8b880f748e9e877a3787aef8a83b896a85c6f2188740b24dfe37c963: Status 404 returned error can't find the container with id f260255f8b880f748e9e877a3787aef8a83b896a85c6f2188740b24dfe37c963
Nov 03 13:13:09.352954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.352927    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"]
Nov 03 13:13:09.359343 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:09.359314    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a0c725_e696_46db_89fa_e022d39904c4.slice/crio-4402e4e97b4765ccf082700a1a90c4ffe837346e649891e99e26844db68686f0 WatchSource:0}: Error finding container 4402e4e97b4765ccf082700a1a90c4ffe837346e649891e99e26844db68686f0: Status 404 returned error can't find the container with id 4402e4e97b4765ccf082700a1a90c4ffe837346e649891e99e26844db68686f0
Nov 03 13:13:09.413822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.413638    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"]
Nov 03 13:13:09.417007 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:09.416966    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c6c8ff_d356_45fa_9642_5d055c0ef0ca.slice/crio-7f61a15bc72650038b9391c6eb707e8135c4cf69daca3329ce914bf67a064689 WatchSource:0}: Error finding container 7f61a15bc72650038b9391c6eb707e8135c4cf69daca3329ce914bf67a064689: Status 404 returned error can't find the container with id 7f61a15bc72650038b9391c6eb707e8135c4cf69daca3329ce914bf67a064689
Nov 03 13:13:09.901312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.901274    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6" event={"ID":"43c6c8ff-d356-45fa-9642-5d055c0ef0ca","Type":"ContainerStarted","Data":"7f61a15bc72650038b9391c6eb707e8135c4cf69daca3329ce914bf67a064689"}
Nov 03 13:13:09.903319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.903290    5420 generic.go:356] "Generic (PLEG): container finished" podID="6808393b-7c55-4452-b277-089ad22d7ae9" containerID="2c2121b778aa6f09e611f9074e66b42f54e01f5bba3bb540e743b94fa3624454" exitCode=0
Nov 03 13:13:09.903448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.903380    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4" event={"ID":"6808393b-7c55-4452-b277-089ad22d7ae9","Type":"ContainerDied","Data":"2c2121b778aa6f09e611f9074e66b42f54e01f5bba3bb540e743b94fa3624454"}
Nov 03 13:13:09.905835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.905808    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn" event={"ID":"26a0c725-e696-46db-89fa-e022d39904c4","Type":"ContainerStarted","Data":"4402e4e97b4765ccf082700a1a90c4ffe837346e649891e99e26844db68686f0"}
Nov 03 13:13:09.909546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:09.909490    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" event={"ID":"8968f39e-7bbd-4d21-afa6-67116c521656","Type":"ContainerStarted","Data":"f260255f8b880f748e9e877a3787aef8a83b896a85c6f2188740b24dfe37c963"}
Nov 03 13:13:10.924326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:10.923303    5420 generic.go:356] "Generic (PLEG): container finished" podID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerID="624545d81d2706a793091a1107e2d025bcfbd55cfc9256fa4d0a717fd2bb1321" exitCode=0
Nov 03 13:13:10.924326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:10.923404    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6" event={"ID":"43c6c8ff-d356-45fa-9642-5d055c0ef0ca","Type":"ContainerDied","Data":"624545d81d2706a793091a1107e2d025bcfbd55cfc9256fa4d0a717fd2bb1321"}
Nov 03 13:13:12.202870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:12.202573    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:12.941290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:12.941247    5420 generic.go:356] "Generic (PLEG): container finished" podID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerID="60744b75028d1f77d9451c22f39c24323cc5b2b98b75520b6254ff8207f59001" exitCode=0
Nov 03 13:13:12.941465 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:12.941310    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6" event={"ID":"43c6c8ff-d356-45fa-9642-5d055c0ef0ca","Type":"ContainerDied","Data":"60744b75028d1f77d9451c22f39c24323cc5b2b98b75520b6254ff8207f59001"}
Nov 03 13:13:13.948973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:13.948804    5420 generic.go:356] "Generic (PLEG): container finished" podID="f7036dbc-3508-410c-8213-41c039b28a7f" containerID="7c3d952fc39ff0bdb46d2b6d72861f03a04ba50fce4b7447992e02ffa768227e" exitCode=0
Nov 03 13:13:13.948973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:13.948887    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d" event={"ID":"f7036dbc-3508-410c-8213-41c039b28a7f","Type":"ContainerDied","Data":"7c3d952fc39ff0bdb46d2b6d72861f03a04ba50fce4b7447992e02ffa768227e"}
Nov 03 13:13:13.955199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:13.955167    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs" event={"ID":"e7006cab-6027-4ad2-9673-58474e10d530","Type":"ContainerStarted","Data":"cbd0a8724ba7ea5a0f7b5d5dd733c9dadd79c35aa84113a4d3e76b90e0b21779"}
Nov 03 13:13:13.958502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:13.958462    5420 generic.go:356] "Generic (PLEG): container finished" podID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerID="52db9f80621eb525521bfe9de53ee3644d7152a9b8efbfa9540314a45c01430b" exitCode=0
Nov 03 13:13:13.958627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:13.958548    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92" event={"ID":"dd04039e-43be-4b17-87d4-326e0f1ac053","Type":"ContainerDied","Data":"52db9f80621eb525521bfe9de53ee3644d7152a9b8efbfa9540314a45c01430b"}
Nov 03 13:13:13.962140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:13.961685    5420 generic.go:356] "Generic (PLEG): container finished" podID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerID="0481190fdab56773e325060ffcdcd8ef10b82f8597d9b333b7ccffd1a51ca258" exitCode=0
Nov 03 13:13:13.962140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:13.961741    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc" event={"ID":"02f77b6e-8c5a-4c8e-9860-f1391bd970fe","Type":"ContainerDied","Data":"0481190fdab56773e325060ffcdcd8ef10b82f8597d9b333b7ccffd1a51ca258"}
Nov 03 13:13:14.970238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:14.970148    5420 generic.go:356] "Generic (PLEG): container finished" podID="6808393b-7c55-4452-b277-089ad22d7ae9" containerID="be6e6a4f3a45ce081ac2aeb165ca6485798882897fe19227ffd37f66601581a6" exitCode=0
Nov 03 13:13:14.970642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:14.970243    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4" event={"ID":"6808393b-7c55-4452-b277-089ad22d7ae9","Type":"ContainerDied","Data":"be6e6a4f3a45ce081ac2aeb165ca6485798882897fe19227ffd37f66601581a6"}
Nov 03 13:13:14.976955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:14.976492    5420 generic.go:356] "Generic (PLEG): container finished" podID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerID="92173011bce2e849216fc049a4a446b2e09fd4664f5a6326a3764564ad7ded24" exitCode=0
Nov 03 13:13:14.976955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:14.976591    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc" event={"ID":"02f77b6e-8c5a-4c8e-9860-f1391bd970fe","Type":"ContainerDied","Data":"92173011bce2e849216fc049a4a446b2e09fd4664f5a6326a3764564ad7ded24"}
Nov 03 13:13:14.980455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:14.980434    5420 generic.go:356] "Generic (PLEG): container finished" podID="e7006cab-6027-4ad2-9673-58474e10d530" containerID="cbd0a8724ba7ea5a0f7b5d5dd733c9dadd79c35aa84113a4d3e76b90e0b21779" exitCode=0
Nov 03 13:13:14.980537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:14.980477    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs" event={"ID":"e7006cab-6027-4ad2-9673-58474e10d530","Type":"ContainerDied","Data":"cbd0a8724ba7ea5a0f7b5d5dd733c9dadd79c35aa84113a4d3e76b90e0b21779"}
Nov 03 13:13:15.440081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.439393    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:13:15.440081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.439484    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:13:15.440081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.439523    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:13:15.529882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.529841    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:13:15.530015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.529936    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:13:15.530015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.529978    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:13:15.989413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.989374    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn" event={"ID":"26a0c725-e696-46db-89fa-e022d39904c4","Type":"ContainerStarted","Data":"6a323dd8c791fdb5ac519092212dcb4408af12e9320ec83177d9453b781faacb"}
Nov 03 13:13:15.991884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.991426    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" event={"ID":"8968f39e-7bbd-4d21-afa6-67116c521656","Type":"ContainerStarted","Data":"8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738"}
Nov 03 13:13:15.993709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.993677    5420 generic.go:356] "Generic (PLEG): container finished" podID="f7036dbc-3508-410c-8213-41c039b28a7f" containerID="ab6b1e4a438adc0901de877b264bc4c435a049e08b5e09a67009ef599a55ac1f" exitCode=0
Nov 03 13:13:15.993841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.993763    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d" event={"ID":"f7036dbc-3508-410c-8213-41c039b28a7f","Type":"ContainerDied","Data":"ab6b1e4a438adc0901de877b264bc4c435a049e08b5e09a67009ef599a55ac1f"}
Nov 03 13:13:15.999273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.999236    5420 generic.go:356] "Generic (PLEG): container finished" podID="e7006cab-6027-4ad2-9673-58474e10d530" containerID="a38d5696681633c6d4b44a725bdbc51703a7bb2f46e0efc5025b0f1452af3667" exitCode=0
Nov 03 13:13:15.999402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:15.999384    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs" event={"ID":"e7006cab-6027-4ad2-9673-58474e10d530","Type":"ContainerDied","Data":"a38d5696681633c6d4b44a725bdbc51703a7bb2f46e0efc5025b0f1452af3667"}
Nov 03 13:13:16.011175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:16.011125    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn" podStartSLOduration=1.84534585 podStartE2EDuration="8.01110956s" totalImagesPullingTime="6.16576371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:16.010461896 +0000 UTC m=+237.088226923" watchObservedRunningTime="2026-11-03 13:13:16.01110956 +0000 UTC m=+237.088874587"
Nov 03 13:13:16.032397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:16.032341    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" podStartSLOduration=1.8900998599999999 podStartE2EDuration="8.032322515s" totalImagesPullingTime="6.142222655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:16.030556725 +0000 UTC m=+237.108321752" watchObservedRunningTime="2026-11-03 13:13:16.032322515 +0000 UTC m=+237.110087542"
Nov 03 13:13:16.057917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:16.057885    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:16.064173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:16.064146    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:16.512964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:16.512924    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:16.963895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:16.963853    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:16.969599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:16.969573    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:17.007085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:17.007052    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:17.007574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:17.007237    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:17.008541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:17.008473    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn"
Nov 03 13:13:17.008865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:17.008834    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:13:19.024018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:19.023974    5420 generic.go:356] "Generic (PLEG): container finished" podID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerID="e5c6b603edeef0f64c5b45e76e8ee6a300f45565aa8d3912d920070eb1200adf" exitCode=0
Nov 03 13:13:19.025290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:19.025264    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92" event={"ID":"dd04039e-43be-4b17-87d4-326e0f1ac053","Type":"ContainerDied","Data":"e5c6b603edeef0f64c5b45e76e8ee6a300f45565aa8d3912d920070eb1200adf"}
Nov 03 13:13:24.745080 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:13:24.744980    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:13:27.093316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:27.092847    5420 generic.go:356] "Generic (PLEG): container finished" podID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerID="7fc28d666d638300c34550094a5cdd71afef6c634174288cf81f215bbb0377de" exitCode=0
Nov 03 13:13:27.093316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:27.092945    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6" event={"ID":"43c6c8ff-d356-45fa-9642-5d055c0ef0ca","Type":"ContainerDied","Data":"7fc28d666d638300c34550094a5cdd71afef6c634174288cf81f215bbb0377de"}
Nov 03 13:13:28.099747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.099710    5420 generic.go:356] "Generic (PLEG): container finished" podID="6808393b-7c55-4452-b277-089ad22d7ae9" containerID="a4397daca15e344a3eeb31a1794127425f1210b2046464f12f905de73e8ee6fa" exitCode=0
Nov 03 13:13:28.100272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.099794    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4" event={"ID":"6808393b-7c55-4452-b277-089ad22d7ae9","Type":"ContainerDied","Data":"a4397daca15e344a3eeb31a1794127425f1210b2046464f12f905de73e8ee6fa"}
Nov 03 13:13:28.102258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.102235    5420 generic.go:356] "Generic (PLEG): container finished" podID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerID="11c9eb4d4e391e3b7421c8a52fa8ee7d855e320e9edbb82addc1f9c940145d3c" exitCode=0
Nov 03 13:13:28.102379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.102319    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92" event={"ID":"dd04039e-43be-4b17-87d4-326e0f1ac053","Type":"ContainerDied","Data":"11c9eb4d4e391e3b7421c8a52fa8ee7d855e320e9edbb82addc1f9c940145d3c"}
Nov 03 13:13:28.104507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.104483    5420 generic.go:356] "Generic (PLEG): container finished" podID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerID="0665fbd21578d7ec66a1419bca47c50db9e08c70ba40ad7425bda2e4569be4b9" exitCode=0
Nov 03 13:13:28.104943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.104887    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc" event={"ID":"02f77b6e-8c5a-4c8e-9860-f1391bd970fe","Type":"ContainerDied","Data":"0665fbd21578d7ec66a1419bca47c50db9e08c70ba40ad7425bda2e4569be4b9"}
Nov 03 13:13:28.237887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.237866    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:28.337691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.337666    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-bundle\") pod \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") "
Nov 03 13:13:28.337813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.337747    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-util\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-util\") pod \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") "
Nov 03 13:13:28.337813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.337786    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-kube-api-access-4jvvr\" (UniqueName: \"kubernetes.io/projected/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-kube-api-access-4jvvr\") pod \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\" (UID: \"43c6c8ff-d356-45fa-9642-5d055c0ef0ca\") "
Nov 03 13:13:28.338141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.338117    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-bundle" pod "43c6c8ff-d356-45fa-9642-5d055c0ef0ca" (UID: "43c6c8ff-d356-45fa-9642-5d055c0ef0ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:28.340194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.340161    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-kube-api-access-4jvvr" pod "43c6c8ff-d356-45fa-9642-5d055c0ef0ca" (UID: "43c6c8ff-d356-45fa-9642-5d055c0ef0ca"). InnerVolumeSpecName "kube-api-access-4jvvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:28.341604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.341580    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-util" pod "43c6c8ff-d356-45fa-9642-5d055c0ef0ca" (UID: "43c6c8ff-d356-45fa-9642-5d055c0ef0ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:28.439042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.439012    5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-util\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:28.439042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.439045    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jvvr\" (UniqueName: \"kubernetes.io/projected/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-kube-api-access-4jvvr\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:28.439230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:28.439062    5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43c6c8ff-d356-45fa-9642-5d055c0ef0ca-bundle\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:29.110487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.110461    5420 generic.go:356] "Generic (PLEG): container finished" podID="f7036dbc-3508-410c-8213-41c039b28a7f" containerID="6e656496ed8fa256ca9548b65b81c96ffa9c9979b1baccba173ceeb64f1aa622" exitCode=0
Nov 03 13:13:29.110904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.110529    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d" event={"ID":"f7036dbc-3508-410c-8213-41c039b28a7f","Type":"ContainerDied","Data":"6e656496ed8fa256ca9548b65b81c96ffa9c9979b1baccba173ceeb64f1aa622"}
Nov 03 13:13:29.112566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.112543    5420 generic.go:356] "Generic (PLEG): container finished" podID="e7006cab-6027-4ad2-9673-58474e10d530" containerID="7f5eb28ecfd0bf254d2495ea4e10479917c8da910e666042d6c1c6a495fbf981" exitCode=0
Nov 03 13:13:29.112683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.112611    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs" event={"ID":"e7006cab-6027-4ad2-9673-58474e10d530","Type":"ContainerDied","Data":"7f5eb28ecfd0bf254d2495ea4e10479917c8da910e666042d6c1c6a495fbf981"}
Nov 03 13:13:29.114368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.114350    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6"
Nov 03 13:13:29.114453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.114397    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ft2k6" event={"ID":"43c6c8ff-d356-45fa-9642-5d055c0ef0ca","Type":"ContainerDied","Data":"7f61a15bc72650038b9391c6eb707e8135c4cf69daca3329ce914bf67a064689"}
Nov 03 13:13:29.114453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.114430    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f61a15bc72650038b9391c6eb707e8135c4cf69daca3329ce914bf67a064689"
Nov 03 13:13:29.259062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.259034    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:29.299405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.299382    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:29.311374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.311347    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:29.347354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.347242    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-util\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-util\") pod \"6808393b-7c55-4452-b277-089ad22d7ae9\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") "
Nov 03 13:13:29.347354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.347313    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6808393b-7c55-4452-b277-089ad22d7ae9-kube-api-access-5pdnt\" (UniqueName: \"kubernetes.io/projected/6808393b-7c55-4452-b277-089ad22d7ae9-kube-api-access-5pdnt\") pod \"6808393b-7c55-4452-b277-089ad22d7ae9\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") "
Nov 03 13:13:29.347354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.347339    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-bundle\") pod \"6808393b-7c55-4452-b277-089ad22d7ae9\" (UID: \"6808393b-7c55-4452-b277-089ad22d7ae9\") "
Nov 03 13:13:29.347805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.347780    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-bundle" pod "6808393b-7c55-4452-b277-089ad22d7ae9" (UID: "6808393b-7c55-4452-b277-089ad22d7ae9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:29.349546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.349519    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6808393b-7c55-4452-b277-089ad22d7ae9-kube-api-access-5pdnt" pod "6808393b-7c55-4452-b277-089ad22d7ae9" (UID: "6808393b-7c55-4452-b277-089ad22d7ae9"). InnerVolumeSpecName "kube-api-access-5pdnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:29.353577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.353551    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-util" pod "6808393b-7c55-4452-b277-089ad22d7ae9" (UID: "6808393b-7c55-4452-b277-089ad22d7ae9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:29.447942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.447908    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-bundle\") pod \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") "
Nov 03 13:13:29.447942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.447945    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-util\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-util\") pod \"dd04039e-43be-4b17-87d4-326e0f1ac053\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") "
Nov 03 13:13:29.448138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.447981    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-kube-api-access-z9g6q\" (UniqueName: \"kubernetes.io/projected/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-kube-api-access-z9g6q\") pod \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") "
Nov 03 13:13:29.448138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.448023    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-bundle\") pod \"dd04039e-43be-4b17-87d4-326e0f1ac053\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") "
Nov 03 13:13:29.448138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.448039    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-util\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-util\") pod \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\" (UID: \"02f77b6e-8c5a-4c8e-9860-f1391bd970fe\") "
Nov 03 13:13:29.448138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.448068    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd04039e-43be-4b17-87d4-326e0f1ac053-kube-api-access-mbz8x\" (UniqueName: \"kubernetes.io/projected/dd04039e-43be-4b17-87d4-326e0f1ac053-kube-api-access-mbz8x\") pod \"dd04039e-43be-4b17-87d4-326e0f1ac053\" (UID: \"dd04039e-43be-4b17-87d4-326e0f1ac053\") "
Nov 03 13:13:29.448385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.448369    5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-bundle\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:29.448443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.448390    5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6808393b-7c55-4452-b277-089ad22d7ae9-util\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:29.448443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.448405    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pdnt\" (UniqueName: \"kubernetes.io/projected/6808393b-7c55-4452-b277-089ad22d7ae9-kube-api-access-5pdnt\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:29.448542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.448487    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-bundle" pod "02f77b6e-8c5a-4c8e-9860-f1391bd970fe" (UID: "02f77b6e-8c5a-4c8e-9860-f1391bd970fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:29.448794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.448769    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-bundle" pod "dd04039e-43be-4b17-87d4-326e0f1ac053" (UID: "dd04039e-43be-4b17-87d4-326e0f1ac053"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:29.450193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.450166    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-kube-api-access-z9g6q" pod "02f77b6e-8c5a-4c8e-9860-f1391bd970fe" (UID: "02f77b6e-8c5a-4c8e-9860-f1391bd970fe"). InnerVolumeSpecName "kube-api-access-z9g6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:29.450562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.450536    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd04039e-43be-4b17-87d4-326e0f1ac053-kube-api-access-mbz8x" pod "dd04039e-43be-4b17-87d4-326e0f1ac053" (UID: "dd04039e-43be-4b17-87d4-326e0f1ac053"). InnerVolumeSpecName "kube-api-access-mbz8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:29.452292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.452266    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-util" pod "02f77b6e-8c5a-4c8e-9860-f1391bd970fe" (UID: "02f77b6e-8c5a-4c8e-9860-f1391bd970fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:29.453184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.453164    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-util" pod "dd04039e-43be-4b17-87d4-326e0f1ac053" (UID: "dd04039e-43be-4b17-87d4-326e0f1ac053"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:29.549513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.549478    5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-bundle\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:29.549513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.549506    5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-util\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:29.549513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.549520    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9g6q\" (UniqueName: \"kubernetes.io/projected/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-kube-api-access-z9g6q\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:29.549746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.549532    5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd04039e-43be-4b17-87d4-326e0f1ac053-bundle\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:29.549746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.549544    5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02f77b6e-8c5a-4c8e-9860-f1391bd970fe-util\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:29.549746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:29.549554    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbz8x\" (UniqueName: \"kubernetes.io/projected/dd04039e-43be-4b17-87d4-326e0f1ac053-kube-api-access-mbz8x\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.119651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.119621    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4"
Nov 03 13:13:30.120130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.119610    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dxs4" event={"ID":"6808393b-7c55-4452-b277-089ad22d7ae9","Type":"ContainerDied","Data":"f32c2f2b5e5073f9744d36ca59e53e60a9b9c4d3ba74ecc33a797bfab27343e9"}
Nov 03 13:13:30.120130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.119748    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f32c2f2b5e5073f9744d36ca59e53e60a9b9c4d3ba74ecc33a797bfab27343e9"
Nov 03 13:13:30.121577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.121548    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92" event={"ID":"dd04039e-43be-4b17-87d4-326e0f1ac053","Type":"ContainerDied","Data":"dd310548d077231bb8aff3e32bea32a36252bcecc125644a7c7ef3ba9f8a5082"}
Nov 03 13:13:30.121694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.121583    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd310548d077231bb8aff3e32bea32a36252bcecc125644a7c7ef3ba9f8a5082"
Nov 03 13:13:30.121694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.121554    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgjd92"
Nov 03 13:13:30.123577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.123560    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc"
Nov 03 13:13:30.123671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.123586    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h66lc" event={"ID":"02f77b6e-8c5a-4c8e-9860-f1391bd970fe","Type":"ContainerDied","Data":"a35b3e36a7c10961f0654f0e996fb2dda181dd3e2c3fd880b9290aa98bd65394"}
Nov 03 13:13:30.123671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.123614    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a35b3e36a7c10961f0654f0e996fb2dda181dd3e2c3fd880b9290aa98bd65394"
Nov 03 13:13:30.247327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.247304    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:30.262561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.262543    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:30.356059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.356033    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-util\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-util\") pod \"f7036dbc-3508-410c-8213-41c039b28a7f\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") "
Nov 03 13:13:30.356238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.356101    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7006cab-6027-4ad2-9673-58474e10d530-kube-api-access-ntqsr\" (UniqueName: \"kubernetes.io/projected/e7006cab-6027-4ad2-9673-58474e10d530-kube-api-access-ntqsr\") pod \"e7006cab-6027-4ad2-9673-58474e10d530\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") "
Nov 03 13:13:30.356238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.356147    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7036dbc-3508-410c-8213-41c039b28a7f-kube-api-access-7qsnf\" (UniqueName: \"kubernetes.io/projected/f7036dbc-3508-410c-8213-41c039b28a7f-kube-api-access-7qsnf\") pod \"f7036dbc-3508-410c-8213-41c039b28a7f\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") "
Nov 03 13:13:30.356238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.356197    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-util\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-util\") pod \"e7006cab-6027-4ad2-9673-58474e10d530\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") "
Nov 03 13:13:30.356402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.356253    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-bundle\") pod \"f7036dbc-3508-410c-8213-41c039b28a7f\" (UID: \"f7036dbc-3508-410c-8213-41c039b28a7f\") "
Nov 03 13:13:30.356402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.356283    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-bundle\") pod \"e7006cab-6027-4ad2-9673-58474e10d530\" (UID: \"e7006cab-6027-4ad2-9673-58474e10d530\") "
Nov 03 13:13:30.356743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.356714    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-bundle" pod "e7006cab-6027-4ad2-9673-58474e10d530" (UID: "e7006cab-6027-4ad2-9673-58474e10d530"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:30.356818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.356775    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-bundle" pod "f7036dbc-3508-410c-8213-41c039b28a7f" (UID: "f7036dbc-3508-410c-8213-41c039b28a7f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:30.358235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.358188    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7006cab-6027-4ad2-9673-58474e10d530-kube-api-access-ntqsr" pod "e7006cab-6027-4ad2-9673-58474e10d530" (UID: "e7006cab-6027-4ad2-9673-58474e10d530"). InnerVolumeSpecName "kube-api-access-ntqsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:30.358351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.358262    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7036dbc-3508-410c-8213-41c039b28a7f-kube-api-access-7qsnf" pod "f7036dbc-3508-410c-8213-41c039b28a7f" (UID: "f7036dbc-3508-410c-8213-41c039b28a7f"). InnerVolumeSpecName "kube-api-access-7qsnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:30.362940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.362915    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-util" pod "e7006cab-6027-4ad2-9673-58474e10d530" (UID: "e7006cab-6027-4ad2-9673-58474e10d530"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:30.363859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.363842    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-util" pod "f7036dbc-3508-410c-8213-41c039b28a7f" (UID: "f7036dbc-3508-410c-8213-41c039b28a7f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:13:30.457385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.457355    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntqsr\" (UniqueName: \"kubernetes.io/projected/e7006cab-6027-4ad2-9673-58474e10d530-kube-api-access-ntqsr\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.457385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.457384    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qsnf\" (UniqueName: \"kubernetes.io/projected/f7036dbc-3508-410c-8213-41c039b28a7f-kube-api-access-7qsnf\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.457556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.457399    5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-util\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.457556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.457410    5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-bundle\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.457556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.457421    5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7006cab-6027-4ad2-9673-58474e10d530-bundle\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.457556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.457432    5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7036dbc-3508-410c-8213-41c039b28a7f-util\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:30.985813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:30.985780    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:31.118155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.118126    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:31.129246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.129206    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d" event={"ID":"f7036dbc-3508-410c-8213-41c039b28a7f","Type":"ContainerDied","Data":"638513b46998094f2bd8b9920cb406d5019161312509fce463315cd9c783d2b2"}
Nov 03 13:13:31.129582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.129246    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bs25d"
Nov 03 13:13:31.129582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.129267    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="638513b46998094f2bd8b9920cb406d5019161312509fce463315cd9c783d2b2"
Nov 03 13:13:31.131472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.131456    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs"
Nov 03 13:13:31.131472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.131469    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448rrbs" event={"ID":"e7006cab-6027-4ad2-9673-58474e10d530","Type":"ContainerDied","Data":"945f45b5e9411d09b907a49e94b58886d26d8e9349212b530713755a814bd61b"}
Nov 03 13:13:31.131640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.131497    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945f45b5e9411d09b907a49e94b58886d26d8e9349212b530713755a814bd61b"
Nov 03 13:13:31.665147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.665113    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:31.763605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763572    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"]
Nov 03 13:13:31.763869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763857    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f7036dbc-3508-410c-8213-41c039b28a7f" containerName="extract"
Nov 03 13:13:31.763919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763871    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7036dbc-3508-410c-8213-41c039b28a7f" containerName="extract"
Nov 03 13:13:31.763919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763883    5420 cpu_manager.go:414] "containerMap: removing container" podUID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerName="util"
Nov 03 13:13:31.763919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763888    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerName="util"
Nov 03 13:13:31.763919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763895    5420 cpu_manager.go:414] "containerMap: removing container" podUID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerName="extract"
Nov 03 13:13:31.763919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763899    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerName="extract"
Nov 03 13:13:31.763919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763910    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e7006cab-6027-4ad2-9673-58474e10d530" containerName="util"
Nov 03 13:13:31.763919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763917    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7006cab-6027-4ad2-9673-58474e10d530" containerName="util"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763924    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e7006cab-6027-4ad2-9673-58474e10d530" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763929    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7006cab-6027-4ad2-9673-58474e10d530" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763934    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e7006cab-6027-4ad2-9673-58474e10d530" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763938    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7006cab-6027-4ad2-9673-58474e10d530" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763949    5420 cpu_manager.go:414] "containerMap: removing container" podUID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763953    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763959    5420 cpu_manager.go:414] "containerMap: removing container" podUID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763963    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763970    5420 cpu_manager.go:414] "containerMap: removing container" podUID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerName="util"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763974    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerName="util"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763980    5420 cpu_manager.go:414] "containerMap: removing container" podUID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763984    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.763990    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f7036dbc-3508-410c-8213-41c039b28a7f" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764002    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7036dbc-3508-410c-8213-41c039b28a7f" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764009    5420 cpu_manager.go:414] "containerMap: removing container" podUID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764015    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764023    5420 cpu_manager.go:414] "containerMap: removing container" podUID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerName="util"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764027    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerName="util"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764032    5420 cpu_manager.go:414] "containerMap: removing container" podUID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764036    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764045    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6808393b-7c55-4452-b277-089ad22d7ae9" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764048    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6808393b-7c55-4452-b277-089ad22d7ae9" containerName="pull"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764054    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6808393b-7c55-4452-b277-089ad22d7ae9" containerName="util"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764057    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6808393b-7c55-4452-b277-089ad22d7ae9" containerName="util"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764067    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f7036dbc-3508-410c-8213-41c039b28a7f" containerName="util"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764071    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7036dbc-3508-410c-8213-41c039b28a7f" containerName="util"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764080    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6808393b-7c55-4452-b277-089ad22d7ae9" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764086    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6808393b-7c55-4452-b277-089ad22d7ae9" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764128    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd04039e-43be-4b17-87d4-326e0f1ac053" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764136    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6808393b-7c55-4452-b277-089ad22d7ae9" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764142    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7036dbc-3508-410c-8213-41c039b28a7f" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764147    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c6c8ff-d356-45fa-9642-5d055c0ef0ca" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764152    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7006cab-6027-4ad2-9673-58474e10d530" containerName="extract"
Nov 03 13:13:31.764161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.764159    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f77b6e-8c5a-4c8e-9860-f1391bd970fe" containerName="extract"
Nov 03 13:13:31.766331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.766308    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.774038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.774016    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 13:13:31.775508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.775491    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 13:13:31.777700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.777678    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 13:13:31.777876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.777862    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 13:13:31.777965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.777954    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dvqfb\""
Nov 03 13:13:31.778000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.777969    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 13:13:31.792336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.792305    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"]
Nov 03 13:13:31.846957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.846926    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:31.868662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.868629    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.868817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.868669    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zpc\" (UniqueName: \"kubernetes.io/projected/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-kube-api-access-77zpc\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.868817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.868752    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-manifests\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.868817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.868803    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.868930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.868822    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.868930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.868852    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-tmp\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.970091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.970066    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.970264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.970100    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zpc\" (UniqueName: \"kubernetes.io/projected/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-kube-api-access-77zpc\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.970264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.970129    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-manifests\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.970264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.970147    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.970264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.970165    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.970264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.970234    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-tmp\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.970570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.970547    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-manifests\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.972635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.972608    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-tmp\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.972765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.972697    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.972832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.972818    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.972944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.972923    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:31.985350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:31.985328    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zpc\" (UniqueName: \"kubernetes.io/projected/63a5d2b7-f0d8-4005-b296-f7707a5b90e6-kube-api-access-77zpc\") pod \"opendatahub-operator-controller-manager-5965948766-45jjk\" (UID: \"63a5d2b7-f0d8-4005-b296-f7707a5b90e6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:32.068007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:32.067970    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:32.080197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:32.080172    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:32.199548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:32.199517    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:32.336744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:32.336678    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:32.381890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:32.381859    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"]
Nov 03 13:13:32.389780 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:32.389752    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a5d2b7_f0d8_4005_b296_f7707a5b90e6.slice/crio-a4be968fef83baa36422212dfc57dd84305a8ca12c762efeff58516492d7cf79 WatchSource:0}: Error finding container a4be968fef83baa36422212dfc57dd84305a8ca12c762efeff58516492d7cf79: Status 404 returned error can't find the container with id a4be968fef83baa36422212dfc57dd84305a8ca12c762efeff58516492d7cf79
Nov 03 13:13:33.144153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:33.144114    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk" event={"ID":"63a5d2b7-f0d8-4005-b296-f7707a5b90e6","Type":"ContainerStarted","Data":"a4be968fef83baa36422212dfc57dd84305a8ca12c762efeff58516492d7cf79"}
Nov 03 13:13:34.788206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:34.788165    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:34.929694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:34.929664    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:35.123060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:35.122987    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:36.160302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:36.160205    5420 generic.go:356] "Generic (PLEG): container finished" podID="63a5d2b7-f0d8-4005-b296-f7707a5b90e6" containerID="08c9b638b8ef2541b3bda43361490b5194370070d7e5827f835c7490e191c378" exitCode=0
Nov 03 13:13:36.160302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:36.160267    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk" event={"ID":"63a5d2b7-f0d8-4005-b296-f7707a5b90e6","Type":"ContainerDied","Data":"08c9b638b8ef2541b3bda43361490b5194370070d7e5827f835c7490e191c378"}
Nov 03 13:13:37.166288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:37.166249    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk" event={"ID":"63a5d2b7-f0d8-4005-b296-f7707a5b90e6","Type":"ContainerStarted","Data":"8c1040538d40cf87140a0f29abbb9092f2a0f3a3acac5ca2ed91deac9738472f"}
Nov 03 13:13:37.166734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:37.166349    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:37.200702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:37.200650    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk" podStartSLOduration=2.341917232 podStartE2EDuration="6.200634888s" totalImagesPullingTime="3.372341706s" totalInitContainerRuntime="486.37595ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:37.199945056 +0000 UTC m=+258.277710082" watchObservedRunningTime="2026-11-03 13:13:37.200634888 +0000 UTC m=+258.278399905"
Nov 03 13:13:39.993501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:39.993455    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:40.126031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:40.126000    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:44.176885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:44.176851    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:44.776432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:44.776391    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:47.663070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.663042    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"]
Nov 03 13:13:47.669194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.669175    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"
Nov 03 13:13:47.678823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.678802    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 13:13:47.679594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.679575    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zrg6f\""
Nov 03 13:13:47.679676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.679634    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"]
Nov 03 13:13:47.679676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.679645    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 13:13:47.681840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.681825    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 13:13:47.804151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.804114    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp9zz\" (UniqueName: \"kubernetes.io/projected/6d127c5c-293c-4c38-8070-20b80d8c5efe-kube-api-access-dp9zz\") pod \"dns-operator-controller-manager-8557cbdb84-t5l28\" (UID: \"6d127c5c-293c-4c38-8070-20b80d8c5efe\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"
Nov 03 13:13:47.905371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.905342    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp9zz\" (UniqueName: \"kubernetes.io/projected/6d127c5c-293c-4c38-8070-20b80d8c5efe-kube-api-access-dp9zz\") pod \"dns-operator-controller-manager-8557cbdb84-t5l28\" (UID: \"6d127c5c-293c-4c38-8070-20b80d8c5efe\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"
Nov 03 13:13:47.921946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.921871    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp9zz\" (UniqueName: \"kubernetes.io/projected/6d127c5c-293c-4c38-8070-20b80d8c5efe-kube-api-access-dp9zz\") pod \"dns-operator-controller-manager-8557cbdb84-t5l28\" (UID: \"6d127c5c-293c-4c38-8070-20b80d8c5efe\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"
Nov 03 13:13:47.982139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:47.982101    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"
Nov 03 13:13:48.172188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:48.172117    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-45jjk"
Nov 03 13:13:48.258274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:48.258246    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"]
Nov 03 13:13:48.260356 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:48.260324    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d127c5c_293c_4c38_8070_20b80d8c5efe.slice/crio-6c7d496a1c305d31baf5783b45d441b72c2a20766128bb8540c554f912192a19 WatchSource:0}: Error finding container 6c7d496a1c305d31baf5783b45d441b72c2a20766128bb8540c554f912192a19: Status 404 returned error can't find the container with id 6c7d496a1c305d31baf5783b45d441b72c2a20766128bb8540c554f912192a19
Nov 03 13:13:49.191832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.191800    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:49.214625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.214591    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28" event={"ID":"6d127c5c-293c-4c38-8070-20b80d8c5efe","Type":"ContainerStarted","Data":"6c7d496a1c305d31baf5783b45d441b72c2a20766128bb8540c554f912192a19"}
Nov 03 13:13:49.375022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.374985    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"]
Nov 03 13:13:49.380552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.380529    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"
Nov 03 13:13:49.397274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.397196    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:13:49.397885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.397856    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 13:13:49.398334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.398316    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9gtkq\""
Nov 03 13:13:49.410512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.410483    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"]
Nov 03 13:13:49.447420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.447345    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:49.518945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.518913    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd327957-66a9-40eb-832b-cb05d0096a36-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mht58\" (UID: \"fd327957-66a9-40eb-832b-cb05d0096a36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"
Nov 03 13:13:49.519125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.518953    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtxnt\" (UniqueName: \"kubernetes.io/projected/fd327957-66a9-40eb-832b-cb05d0096a36-kube-api-access-dtxnt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mht58\" (UID: \"fd327957-66a9-40eb-832b-cb05d0096a36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"
Nov 03 13:13:49.620399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.620358    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd327957-66a9-40eb-832b-cb05d0096a36-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mht58\" (UID: \"fd327957-66a9-40eb-832b-cb05d0096a36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"
Nov 03 13:13:49.620551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.620413    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxnt\" (UniqueName: \"kubernetes.io/projected/fd327957-66a9-40eb-832b-cb05d0096a36-kube-api-access-dtxnt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mht58\" (UID: \"fd327957-66a9-40eb-832b-cb05d0096a36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"
Nov 03 13:13:49.620970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.620953    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd327957-66a9-40eb-832b-cb05d0096a36-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mht58\" (UID: \"fd327957-66a9-40eb-832b-cb05d0096a36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"
Nov 03 13:13:49.632925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.632899    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxnt\" (UniqueName: \"kubernetes.io/projected/fd327957-66a9-40eb-832b-cb05d0096a36-kube-api-access-dtxnt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mht58\" (UID: \"fd327957-66a9-40eb-832b-cb05d0096a36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"
Nov 03 13:13:49.696518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:49.696490    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"
Nov 03 13:13:50.002227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:50.002139    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58"]
Nov 03 13:13:50.005728 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:50.005695    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd327957_66a9_40eb_832b_cb05d0096a36.slice/crio-b6d6d7514a536e3bcf4997113f1d7867b88f53b3ba37fb4915379177cce17f9a WatchSource:0}: Error finding container b6d6d7514a536e3bcf4997113f1d7867b88f53b3ba37fb4915379177cce17f9a: Status 404 returned error can't find the container with id b6d6d7514a536e3bcf4997113f1d7867b88f53b3ba37fb4915379177cce17f9a
Nov 03 13:13:50.221129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:50.221094    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58" event={"ID":"fd327957-66a9-40eb-832b-cb05d0096a36","Type":"ContainerStarted","Data":"b6d6d7514a536e3bcf4997113f1d7867b88f53b3ba37fb4915379177cce17f9a"}
Nov 03 13:13:51.204015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.203964    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:51.229640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.229168    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28" event={"ID":"6d127c5c-293c-4c38-8070-20b80d8c5efe","Type":"ContainerStarted","Data":"56e0549ecf160dd14b4d68ebf96c5898192b4ac9fe9ded7b65b4d27a277fd2ee"}
Nov 03 13:13:51.229640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.229286    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"
Nov 03 13:13:51.249826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.249776    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28" podStartSLOduration=1.4005047959999999 podStartE2EDuration="4.249762341s" totalImagesPullingTime="2.849257545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:51.248527338 +0000 UTC m=+272.326292363" watchObservedRunningTime="2026-11-03 13:13:51.249762341 +0000 UTC m=+272.327527368"
Nov 03 13:13:51.486584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.486549    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"]
Nov 03 13:13:51.490317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.490285    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"
Nov 03 13:13:51.514449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.514423    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qbcgv\""
Nov 03 13:13:51.517508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.517477    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"]
Nov 03 13:13:51.637182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.637136    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqxtx\" (UniqueName: \"kubernetes.io/projected/440de568-380c-4201-95bc-d07af7d3d473-kube-api-access-vqxtx\") pod \"limitador-operator-controller-manager-5778bb4449-tzjk2\" (UID: \"440de568-380c-4201-95bc-d07af7d3d473\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"
Nov 03 13:13:51.738439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.738363    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqxtx\" (UniqueName: \"kubernetes.io/projected/440de568-380c-4201-95bc-d07af7d3d473-kube-api-access-vqxtx\") pod \"limitador-operator-controller-manager-5778bb4449-tzjk2\" (UID: \"440de568-380c-4201-95bc-d07af7d3d473\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"
Nov 03 13:13:51.751507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.751479    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqxtx\" (UniqueName: \"kubernetes.io/projected/440de568-380c-4201-95bc-d07af7d3d473-kube-api-access-vqxtx\") pod \"limitador-operator-controller-manager-5778bb4449-tzjk2\" (UID: \"440de568-380c-4201-95bc-d07af7d3d473\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"
Nov 03 13:13:51.790152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.790121    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:51.804124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.804099    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"
Nov 03 13:13:51.923503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:51.923471    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:52.089811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:52.089739    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"]
Nov 03 13:13:52.093118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:52.093094    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440de568_380c_4201_95bc_d07af7d3d473.slice/crio-bacd9474bd6d5d21127467e7513626c05777dd6fbba43f985b90846e98f1b5b2 WatchSource:0}: Error finding container bacd9474bd6d5d21127467e7513626c05777dd6fbba43f985b90846e98f1b5b2: Status 404 returned error can't find the container with id bacd9474bd6d5d21127467e7513626c05777dd6fbba43f985b90846e98f1b5b2
Nov 03 13:13:52.235375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:52.235334    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58" event={"ID":"fd327957-66a9-40eb-832b-cb05d0096a36","Type":"ContainerStarted","Data":"f4314733ef4b3cd8cfab0c715a3d9e52896d23da515d5bbcbcb820cf06a5ed50"}
Nov 03 13:13:52.236672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:52.236649    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2" event={"ID":"440de568-380c-4201-95bc-d07af7d3d473","Type":"ContainerStarted","Data":"bacd9474bd6d5d21127467e7513626c05777dd6fbba43f985b90846e98f1b5b2"}
Nov 03 13:13:52.261855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:52.261797    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mht58" podStartSLOduration=1.446386063 podStartE2EDuration="3.261779711s" totalImagesPullingTime="1.815393648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:52.260804012 +0000 UTC m=+273.338569045" watchObservedRunningTime="2026-11-03 13:13:52.261779711 +0000 UTC m=+273.339544739"
Nov 03 13:13:52.375493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:52.375419    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:53.245470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:53.245438    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:53.838966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:53.838932    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:54.020528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:54.020387    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:54.248515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:54.248484    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2" event={"ID":"440de568-380c-4201-95bc-d07af7d3d473","Type":"ContainerStarted","Data":"d6adeb4a8084bdafaa48bacd1e217bb51bbfc9c9894a6f899c4779e767aa8197"}
Nov 03 13:13:54.248941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:54.248557    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"
Nov 03 13:13:54.276056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:54.275998    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2" podStartSLOduration=1.366025223 podStartE2EDuration="3.275984217s" totalImagesPullingTime="1.909958994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:54.274860547 +0000 UTC m=+275.352625572" watchObservedRunningTime="2026-11-03 13:13:54.275984217 +0000 UTC m=+275.353749243"
Nov 03 13:13:54.729927 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:13:54.729888    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:13:55.713767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:55.713737    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"]
Nov 03 13:13:55.717075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:55.717060    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"
Nov 03 13:13:55.737473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:55.737200    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fkpvq\""
Nov 03 13:13:55.739393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:55.739368    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"]
Nov 03 13:13:55.873989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:55.873956    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2zx\" (UniqueName: \"kubernetes.io/projected/d8e7799f-b83f-48ff-b7c5-3baa8242f13e-kube-api-access-6g2zx\") pod \"authorino-operator-7b7bb4bc77-p5jpw\" (UID: \"d8e7799f-b83f-48ff-b7c5-3baa8242f13e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"
Nov 03 13:13:55.975047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:55.974974    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2zx\" (UniqueName: \"kubernetes.io/projected/d8e7799f-b83f-48ff-b7c5-3baa8242f13e-kube-api-access-6g2zx\") pod \"authorino-operator-7b7bb4bc77-p5jpw\" (UID: \"d8e7799f-b83f-48ff-b7c5-3baa8242f13e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"
Nov 03 13:13:55.984376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:55.984349    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2zx\" (UniqueName: \"kubernetes.io/projected/d8e7799f-b83f-48ff-b7c5-3baa8242f13e-kube-api-access-6g2zx\") pod \"authorino-operator-7b7bb4bc77-p5jpw\" (UID: \"d8e7799f-b83f-48ff-b7c5-3baa8242f13e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"
Nov 03 13:13:56.029186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:56.029157    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"
Nov 03 13:13:56.102467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:56.102436    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:56.250101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:56.250017    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:56.301708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:56.301685    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"]
Nov 03 13:13:56.304362 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:13:56.304334    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e7799f_b83f_48ff_b7c5_3baa8242f13e.slice/crio-85e710b746360c92289264497d09d88777f5187c6a7ffb6351ada3845c37e9b2 WatchSource:0}: Error finding container 85e710b746360c92289264497d09d88777f5187c6a7ffb6351ada3845c37e9b2: Status 404 returned error can't find the container with id 85e710b746360c92289264497d09d88777f5187c6a7ffb6351ada3845c37e9b2
Nov 03 13:13:56.578504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:56.578410    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:57.262874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:57.262843    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw" event={"ID":"d8e7799f-b83f-48ff-b7c5-3baa8242f13e","Type":"ContainerStarted","Data":"85e710b746360c92289264497d09d88777f5187c6a7ffb6351ada3845c37e9b2"}
Nov 03 13:13:57.372508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:57.372471    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:58.195386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:58.195317    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:59.272562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:59.272527    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw" event={"ID":"d8e7799f-b83f-48ff-b7c5-3baa8242f13e","Type":"ContainerStarted","Data":"1d2e66a5612162cc542821a374c6867a477c36ca5a6137ff7b929264176639c6"}
Nov 03 13:13:59.273023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:59.272658    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"
Nov 03 13:13:59.298499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:59.298448    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw" podStartSLOduration=2.129243744 podStartE2EDuration="4.298435141s" totalImagesPullingTime="2.169191397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:59.296175702 +0000 UTC m=+280.373940728" watchObservedRunningTime="2026-11-03 13:13:59.298435141 +0000 UTC m=+280.376200166"
Nov 03 13:13:59.881459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:13:59.881425    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:00.138893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:00.138809    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:00.973851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:00.973815    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:01.107424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:01.107393    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:02.239344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:02.239316    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t5l28"
Nov 03 13:14:02.766979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:02.766949    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:02.898769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:02.898733    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:03.446673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.446640    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"]
Nov 03 13:14:03.450287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.450260    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:03.460939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.460914    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5qr2g\""
Nov 03 13:14:03.461484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.461462    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"]
Nov 03 13:14:03.563996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.563963    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:03.635801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.635765    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92636d04-16fd-4272-af4a-505c26c9ae37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wpzv6\" (UID: \"92636d04-16fd-4272-af4a-505c26c9ae37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:03.635970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.635818    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r95nm\" (UniqueName: \"kubernetes.io/projected/92636d04-16fd-4272-af4a-505c26c9ae37-kube-api-access-r95nm\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wpzv6\" (UID: \"92636d04-16fd-4272-af4a-505c26c9ae37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:03.737117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.737084    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92636d04-16fd-4272-af4a-505c26c9ae37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wpzv6\" (UID: \"92636d04-16fd-4272-af4a-505c26c9ae37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:03.737326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.737161    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r95nm\" (UniqueName: \"kubernetes.io/projected/92636d04-16fd-4272-af4a-505c26c9ae37-kube-api-access-r95nm\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wpzv6\" (UID: \"92636d04-16fd-4272-af4a-505c26c9ae37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:03.737502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.737482    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92636d04-16fd-4272-af4a-505c26c9ae37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wpzv6\" (UID: \"92636d04-16fd-4272-af4a-505c26c9ae37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:03.758109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.758067    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:03.760001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.759974    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r95nm\" (UniqueName: \"kubernetes.io/projected/92636d04-16fd-4272-af4a-505c26c9ae37-kube-api-access-r95nm\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wpzv6\" (UID: \"92636d04-16fd-4272-af4a-505c26c9ae37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:03.765569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.765548    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:03.897128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:03.897089    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:04.041476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:04.041404    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"]
Nov 03 13:14:04.046783 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:14:04.046757    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92636d04_16fd_4272_af4a_505c26c9ae37.slice/crio-b9e1ab6e0b0f9b6c2d74f3bee4140ae122c79725a8e92c68573907e368149555 WatchSource:0}: Error finding container b9e1ab6e0b0f9b6c2d74f3bee4140ae122c79725a8e92c68573907e368149555: Status 404 returned error can't find the container with id b9e1ab6e0b0f9b6c2d74f3bee4140ae122c79725a8e92c68573907e368149555
Nov 03 13:14:04.240056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:04.240024    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:04.297402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:04.297329    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6" event={"ID":"92636d04-16fd-4272-af4a-505c26c9ae37","Type":"ContainerStarted","Data":"b9e1ab6e0b0f9b6c2d74f3bee4140ae122c79725a8e92c68573907e368149555"}
Nov 03 13:14:05.254883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:05.254848    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tzjk2"
Nov 03 13:14:05.986618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:05.986583    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:06.143505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:06.143475    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:08.305149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.305062    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:08.320673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.320637    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6" event={"ID":"92636d04-16fd-4272-af4a-505c26c9ae37","Type":"ContainerStarted","Data":"b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1"}
Nov 03 13:14:08.320862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.320743    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:08.349684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.349629    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6" podStartSLOduration=1.350116573 podStartE2EDuration="5.349616368s" totalImagesPullingTime="3.999499795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:08.347406953 +0000 UTC m=+289.425171979" watchObservedRunningTime="2026-11-03 13:14:08.349616368 +0000 UTC m=+289.427381394"
Nov 03 13:14:08.468606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.468572    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:08.640502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.640411    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:08.777981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.777950    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"]
Nov 03 13:14:08.781520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.781501    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.791169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.791146    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7kwbn\""
Nov 03 13:14:08.795844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.795820    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"]
Nov 03 13:14:08.813463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.813431    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:08.876985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.876945    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.877163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.876996    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.877163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.877064    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.877163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.877129    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.877307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.877182    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.877307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.877258    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwsz\" (UniqueName: \"kubernetes.io/projected/f2205642-0a8a-457f-862a-f86a9176d5e5-kube-api-access-hzwsz\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.877307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.877293    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.877400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.877323    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.877400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.877356    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2205642-0a8a-457f-862a-f86a9176d5e5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.968818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.968790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:08.977938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.977911    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.977946    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.977968    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.977987    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978021    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978050    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsz\" (UniqueName: \"kubernetes.io/projected/f2205642-0a8a-457f-862a-f86a9176d5e5-kube-api-access-hzwsz\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978070    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978099    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978132    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2205642-0a8a-457f-862a-f86a9176d5e5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978425    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978472    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978521    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978769    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.978938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.978901    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2205642-0a8a-457f-862a-f86a9176d5e5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.980588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.980570    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:08.980706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:08.980689    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:09.009493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:09.009459    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2205642-0a8a-457f-862a-f86a9176d5e5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:09.009646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:09.009628    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwsz\" (UniqueName: \"kubernetes.io/projected/f2205642-0a8a-457f-862a-f86a9176d5e5-kube-api-access-hzwsz\") pod \"data-science-gateway-data-science-gateway-class-7b67449fb5mtj65\" (UID: \"f2205642-0a8a-457f-862a-f86a9176d5e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:09.093968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:09.093935    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:09.219896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:09.219737    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:09.415906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:09.415880    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"]
Nov 03 13:14:09.418756 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:14:09.418727    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2205642_0a8a_457f_862a_f86a9176d5e5.slice/crio-eccc06934f822345629a2023fbbd1d7deef71744a3bccf75e3827af06e30b675 WatchSource:0}: Error finding container eccc06934f822345629a2023fbbd1d7deef71744a3bccf75e3827af06e30b675: Status 404 returned error can't find the container with id eccc06934f822345629a2023fbbd1d7deef71744a3bccf75e3827af06e30b675
Nov 03 13:14:09.421511 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:09.421439    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:14:09.421713 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:09.421618    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:14:09.421892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:09.421741    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:14:10.033853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:10.033818    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:10.281022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:10.280994    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p5jpw"
Nov 03 13:14:10.333133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:10.333038    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65" event={"ID":"f2205642-0a8a-457f-862a-f86a9176d5e5","Type":"ContainerStarted","Data":"fb823263db689fa7b6e3bc97fd8352e9b1a74b4c185e03c8c21f0d697a14d6df"}
Nov 03 13:14:10.333133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:10.333094    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65" event={"ID":"f2205642-0a8a-457f-862a-f86a9176d5e5","Type":"ContainerStarted","Data":"eccc06934f822345629a2023fbbd1d7deef71744a3bccf75e3827af06e30b675"}
Nov 03 13:14:10.359075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:10.359024    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65" podStartSLOduration=2.359007841 podStartE2EDuration="2.359007841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:10.357242693 +0000 UTC m=+291.435007718" watchObservedRunningTime="2026-11-03 13:14:10.359007841 +0000 UTC m=+291.436772867"
Nov 03 13:14:10.593760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:10.593685    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:10.795705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:10.795671    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:10.965997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:10.965953    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:11.095030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.094996    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:11.100463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.100437    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:11.269603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.269515    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:14:11.277621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.277593    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:11.284884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.284853    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lsg7c\""
Nov 03 13:14:11.286430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.286409    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:14:11.297483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.297459    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63-config-file\") pod \"limitador-limitador-7c9b87d574-fhvn2\" (UID: \"d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:11.297622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.297504    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmh2\" (UniqueName: \"kubernetes.io/projected/d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63-kube-api-access-rrmh2\") pod \"limitador-limitador-7c9b87d574-fhvn2\" (UID: \"d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:11.314616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.314584    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 13:14:11.334136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.334098    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:14:11.338827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.338795    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:11.339890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.339864    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7b67449fb5mtj65"
Nov 03 13:14:11.391441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.391406    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"]
Nov 03 13:14:11.395960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.395934    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:11.398890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.398861    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dccx\" (UniqueName: \"kubernetes.io/projected/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-kube-api-access-8dccx\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:11.399039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.398924    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63-config-file\") pod \"limitador-limitador-7c9b87d574-fhvn2\" (UID: \"d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:11.399039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.398966    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmh2\" (UniqueName: \"kubernetes.io/projected/d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63-kube-api-access-rrmh2\") pod \"limitador-limitador-7c9b87d574-fhvn2\" (UID: \"d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:11.399039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.399010    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:11.399202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.399089    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:11.400941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.400912    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63-config-file\") pod \"limitador-limitador-7c9b87d574-fhvn2\" (UID: \"d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:11.404763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.404732    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"]
Nov 03 13:14:11.404962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.404947    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 13:14:11.413039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.413014    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmh2\" (UniqueName: \"kubernetes.io/projected/d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63-kube-api-access-rrmh2\") pod \"limitador-limitador-7c9b87d574-fhvn2\" (UID: \"d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:11.414667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.414644    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 13:14:11.457445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.457410    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:11.499856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.499826    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:11.500036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.499874    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dccx\" (UniqueName: \"kubernetes.io/projected/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-kube-api-access-8dccx\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:11.500036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.499911    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:11.500116 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:14:11.500051    5420 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Nov 03 13:14:11.500151 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:14:11.500133    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-plugin-serving-cert podName:bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef nodeName:}" failed. No retries permitted until 2026-11-03 13:14:12.00011325 +0000 UTC m=+293.077878269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-n87db" (UID: "bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef") : secret "plugin-serving-cert" not found
Nov 03 13:14:11.500550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.500534    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:11.510918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.510888    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dccx\" (UniqueName: \"kubernetes.io/projected/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-kube-api-access-8dccx\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:11.592794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.592719    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:11.677679 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.674915    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:11.828087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.828054    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:11.896737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:11.896595    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:14:11.899191 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:14:11.899164    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ef8cc9_9fc6_4fae_9efb_c9043ab9fe63.slice/crio-b4a1e21bb6dc843760cdfddf66b85e89511d7f60d5bbd183c4e9a35c76664456 WatchSource:0}: Error finding container b4a1e21bb6dc843760cdfddf66b85e89511d7f60d5bbd183c4e9a35c76664456: Status 404 returned error can't find the container with id b4a1e21bb6dc843760cdfddf66b85e89511d7f60d5bbd183c4e9a35c76664456
Nov 03 13:14:12.002483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:12.002447    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:12.005054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:12.005032    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n87db\" (UID: \"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:12.029673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:12.029644    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"
Nov 03 13:14:12.303355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:12.303324    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db"]
Nov 03 13:14:12.306479 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:14:12.306447    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc8f05c9_fb1e_4ff4_b06c_b4ee227119ef.slice/crio-62f4c0fd2404a42cf6f66cb22b7ec48faf68b7814053ac962492efe8421b94c8 WatchSource:0}: Error finding container 62f4c0fd2404a42cf6f66cb22b7ec48faf68b7814053ac962492efe8421b94c8: Status 404 returned error can't find the container with id 62f4c0fd2404a42cf6f66cb22b7ec48faf68b7814053ac962492efe8421b94c8
Nov 03 13:14:12.343929 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:12.343891    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db" event={"ID":"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef","Type":"ContainerStarted","Data":"62f4c0fd2404a42cf6f66cb22b7ec48faf68b7814053ac962492efe8421b94c8"}
Nov 03 13:14:12.344967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:12.344941    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2" event={"ID":"d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63","Type":"ContainerStarted","Data":"b4a1e21bb6dc843760cdfddf66b85e89511d7f60d5bbd183c4e9a35c76664456"}
Nov 03 13:14:13.355277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:13.355181    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2" event={"ID":"d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63","Type":"ContainerStarted","Data":"74a77eef92bd3d4630ce54d44653c2815ff8256b03278bbb9c0a1f0932b84835"}
Nov 03 13:14:13.355277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:13.355275    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:13.374704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:13.374642    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2" podStartSLOduration=1.177172713 podStartE2EDuration="2.374619754s" totalImagesPullingTime="1.197447041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:13.372615992 +0000 UTC m=+294.450381043" watchObservedRunningTime="2026-11-03 13:14:13.374619754 +0000 UTC m=+294.452384781"
Nov 03 13:14:14.564325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:14.564292    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:14.705190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:14.705151    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:14.922147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:14.922072    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:16.572535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:16.572499    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:17.376703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:17.376668    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db" event={"ID":"bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef","Type":"ContainerStarted","Data":"2be3762f76d56adb06faeeccf61cbd96169ab1e9db0de4299779ba5f037a1e51"}
Nov 03 13:14:17.393327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:17.393279    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n87db" podStartSLOduration=1.479677919 podStartE2EDuration="6.393265096s" totalImagesPullingTime="4.913587177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:17.392110093 +0000 UTC m=+298.469875122" watchObservedRunningTime="2026-11-03 13:14:17.393265096 +0000 UTC m=+298.471030121"
Nov 03 13:14:19.107943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:19.107918    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:14:19.108407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:19.107918    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:14:19.110493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:19.110468    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:19.238722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:19.238688    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:19.327831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:19.327799    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:24.730684 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:14:24.730641    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:14:25.366721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:25.366689    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"
Nov 03 13:14:25.796271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:25.796235    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:25.980893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:25.980861    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:28.984491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:28.984454    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:29.169394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:29.169364    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:30.871061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:30.871015    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:31.095732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:31.095694    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:31.668039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:31.668002    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:33.397961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:33.397923    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:33.534140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:33.534111    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:35.626297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:35.626264    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"]
Nov 03 13:14:35.626713 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:35.626474    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6" podUID="92636d04-16fd-4272-af4a-505c26c9ae37" containerName="manager" containerID="cri-o://b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1" gracePeriod=10
Nov 03 13:14:35.865343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:35.865320    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:35.920898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:35.920822    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92636d04-16fd-4272-af4a-505c26c9ae37-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92636d04-16fd-4272-af4a-505c26c9ae37-extensions-socket-volume\") pod \"92636d04-16fd-4272-af4a-505c26c9ae37\" (UID: \"92636d04-16fd-4272-af4a-505c26c9ae37\") "
Nov 03 13:14:35.921075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:35.920897    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92636d04-16fd-4272-af4a-505c26c9ae37-kube-api-access-r95nm\" (UniqueName: \"kubernetes.io/projected/92636d04-16fd-4272-af4a-505c26c9ae37-kube-api-access-r95nm\") pod \"92636d04-16fd-4272-af4a-505c26c9ae37\" (UID: \"92636d04-16fd-4272-af4a-505c26c9ae37\") "
Nov 03 13:14:35.921278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:35.921251    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92636d04-16fd-4272-af4a-505c26c9ae37-extensions-socket-volume" pod "92636d04-16fd-4272-af4a-505c26c9ae37" (UID: "92636d04-16fd-4272-af4a-505c26c9ae37"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:35.923121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:35.923091    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92636d04-16fd-4272-af4a-505c26c9ae37-kube-api-access-r95nm" pod "92636d04-16fd-4272-af4a-505c26c9ae37" (UID: "92636d04-16fd-4272-af4a-505c26c9ae37"). InnerVolumeSpecName "kube-api-access-r95nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:36.022139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.022101    5420 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92636d04-16fd-4272-af4a-505c26c9ae37-extensions-socket-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:36.022139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.022132    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r95nm\" (UniqueName: \"kubernetes.io/projected/92636d04-16fd-4272-af4a-505c26c9ae37-kube-api-access-r95nm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:36.464147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.464107    5420 generic.go:356] "Generic (PLEG): container finished" podID="92636d04-16fd-4272-af4a-505c26c9ae37" containerID="b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1" exitCode=0
Nov 03 13:14:36.464365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.464173    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6" event={"ID":"92636d04-16fd-4272-af4a-505c26c9ae37","Type":"ContainerDied","Data":"b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1"}
Nov 03 13:14:36.464365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.464183    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"
Nov 03 13:14:36.464365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.464286    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6" event={"ID":"92636d04-16fd-4272-af4a-505c26c9ae37","Type":"ContainerDied","Data":"b9e1ab6e0b0f9b6c2d74f3bee4140ae122c79725a8e92c68573907e368149555"}
Nov 03 13:14:36.464365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.464317    5420 scope.go:122] "RemoveContainer" containerID="b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1"
Nov 03 13:14:36.474314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.474293    5420 scope.go:122] "RemoveContainer" containerID="b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1"
Nov 03 13:14:36.474604 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:14:36.474580    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1\": container with ID starting with b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1 not found: ID does not exist" containerID="b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1"
Nov 03 13:14:36.474688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.474616    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1"} err="failed to get container status \"b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1\": rpc error: code = NotFound desc = could not find container \"b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1\": container with ID starting with b64bb949f3bee25684e822808b99ac737df5725390543e2ebeebac31b259c1c1 not found: ID does not exist"
Nov 03 13:14:36.515968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.515935    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"]
Nov 03 13:14:36.522173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:36.522146    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wpzv6"]
Nov 03 13:14:37.082076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:37.082045    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92636d04-16fd-4272-af4a-505c26c9ae37" path="/var/lib/kubelet/pods/92636d04-16fd-4272-af4a-505c26c9ae37/volumes"
Nov 03 13:14:37.982547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:37.982501    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:38.196401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.196370    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:38.365130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.365056    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:38.371721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.371693    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"]
Nov 03 13:14:38.372249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.372200    5420 cpu_manager.go:414] "containerMap: removing container" podUID="92636d04-16fd-4272-af4a-505c26c9ae37" containerName="manager"
Nov 03 13:14:38.372408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.372251    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92636d04-16fd-4272-af4a-505c26c9ae37" containerName="manager"
Nov 03 13:14:38.372459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.372416    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="92636d04-16fd-4272-af4a-505c26c9ae37" containerName="manager"
Nov 03 13:14:38.378026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.378007    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.384383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.384360    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 13:14:38.384521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.384447    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-zfnch\""
Nov 03 13:14:38.384521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.384473    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 13:14:38.384521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.384490    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 13:14:38.384672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.384557    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 13:14:38.384672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.384648    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 13:14:38.386729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.386706    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"]
Nov 03 13:14:38.445362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.445322    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-manager-config\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.445362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.445365    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-cert\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.445583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.445471    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-metrics-certs\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.445583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.445499    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfs8g\" (UniqueName: \"kubernetes.io/projected/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-kube-api-access-vfs8g\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.497500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.497461    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:38.546645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.546602    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-manager-config\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.546645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.546646    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-cert\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.546903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.546685    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-metrics-certs\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.546903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.546707    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfs8g\" (UniqueName: \"kubernetes.io/projected/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-kube-api-access-vfs8g\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.547328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.547288    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-manager-config\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.549205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.549183    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-metrics-certs\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.549300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.549238    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-cert\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.555976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.555956    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfs8g\" (UniqueName: \"kubernetes.io/projected/da98a3b0-5b57-41e3-aa0a-e19a509a9c08-kube-api-access-vfs8g\") pod \"jobset-controller-manager-fdc87559d-vj28t\" (UID: \"da98a3b0-5b57-41e3-aa0a-e19a509a9c08\") " pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.691764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.691679    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:38.970842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.970708    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"]
Nov 03 13:14:38.973026 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:14:38.972991    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda98a3b0_5b57_41e3_aa0a_e19a509a9c08.slice/crio-b7a62f50283d8a490b9e2daa6202f1f54008b61cdea00c5592fdd14fae6f54cc WatchSource:0}: Error finding container b7a62f50283d8a490b9e2daa6202f1f54008b61cdea00c5592fdd14fae6f54cc: Status 404 returned error can't find the container with id b7a62f50283d8a490b9e2daa6202f1f54008b61cdea00c5592fdd14fae6f54cc
Nov 03 13:14:38.975134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:38.975115    5420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:14:39.480485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:39.480437    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t" event={"ID":"da98a3b0-5b57-41e3-aa0a-e19a509a9c08","Type":"ContainerStarted","Data":"b7a62f50283d8a490b9e2daa6202f1f54008b61cdea00c5592fdd14fae6f54cc"}
Nov 03 13:14:42.159295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:42.159260    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:42.323024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:42.322989    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:42.499302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:42.499268    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t" event={"ID":"da98a3b0-5b57-41e3-aa0a-e19a509a9c08","Type":"ContainerStarted","Data":"ae834018b81e5a7c012c3e411d08ec8a20291d7ac04239e8e47467d0e144a1d6"}
Nov 03 13:14:42.499513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:42.499395    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:42.522015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:42.521959    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t" podStartSLOduration=1.328444281 podStartE2EDuration="4.521945713s" totalImagesPullingTime="3.193501432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:42.519449227 +0000 UTC m=+323.597214253" watchObservedRunningTime="2026-11-03 13:14:42.521945713 +0000 UTC m=+323.599710738"
Nov 03 13:14:50.911692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:50.911658    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:51.041667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:51.041634    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:51.168481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:51.168403    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:53.470790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:53.470757    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:53.509903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:53.509874    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-fdc87559d-vj28t"
Nov 03 13:14:53.593747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:53.593721    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:54.730138 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:14:54.730104    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:14:55.895463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:55.895432    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:56.028264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:56.028229    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:56.421978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:56.421942    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:56.947940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:56.947910    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:57.080141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:14:57.080106    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:05.236066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:05.236035    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:05.397336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:05.397303    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:05.551857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:05.551774    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:05.808080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:05.808001    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:05.940870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:05.940841    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:06.521514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:06.521483    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:06.654970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:06.654938    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:06.787868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:06.787790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:16.610550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:16.610506    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:19.848807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:19.848770    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:19.973537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:19.973503    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:23.694112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:23.694038    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:23.838447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:23.838410    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:24.729723 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:15:24.729682    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:15:24.991028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:24.990952    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:25.133460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:25.133423    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:25.278943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:25.278866    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:25.829844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:25.829809    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:15:33.047414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:33.047368    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6sg42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:15:33.047874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:33.047456    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:15:35.764165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:35.764131    5420 generic.go:356] "Generic (PLEG): container finished" podID="8968f39e-7bbd-4d21-afa6-67116c521656" containerID="8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738" exitCode=0
Nov 03 13:15:35.764663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:35.764175    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" event={"ID":"8968f39e-7bbd-4d21-afa6-67116c521656","Type":"ContainerDied","Data":"8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738"}
Nov 03 13:15:35.764663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:35.764526    5420 scope.go:122] "RemoveContainer" containerID="8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738"
Nov 03 13:15:35.765594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:35.765563    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:15:35.765654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:35.765641    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:15:35.765689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:35.765676    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:15:36.770769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:36.770739    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" event={"ID":"8968f39e-7bbd-4d21-afa6-67116c521656","Type":"ContainerStarted","Data":"34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03"}
Nov 03 13:15:36.963412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:36.963375    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:15:36.963635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:36.963609    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6sg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Nov 03 13:15:36.963691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:36.963670    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Nov 03 13:15:37.845915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:37.845872    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:37.963327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:37.963297    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6sg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Nov 03 13:15:37.963541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:37.963521    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Nov 03 13:15:37.967836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:37.967786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:38.963744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:38.963711    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6sg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Nov 03 13:15:38.964147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:38.963768    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Nov 03 13:15:39.964045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:39.964010    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6sg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Nov 03 13:15:39.964479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:39.964067    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Nov 03 13:15:40.963618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:40.963586    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6sg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Nov 03 13:15:40.963811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:40.963648    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Nov 03 13:15:41.966918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:41.966886    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:15:41.967416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:41.967310    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:15:41.968023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:15:41.968004    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:15:54.729894 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:15:54.729857    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:16:16.674577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:16:16.674538    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:24.730294 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:16:24.730258    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:16:36.540476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:16:36.540442    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:36.689318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:16:36.689285    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:40.304960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:16:40.304932    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:40.445265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:16:40.445232    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:45.485421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:16:45.485386    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:45.627242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:16:45.627192    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:54.729162 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:16:54.729121    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:17:05.074133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:05.074046    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:05.624541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:05.624512    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:07.554602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:07.554570    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:07.717098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:07.717066    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:11.946949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:11.946915    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:12.302718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:12.302684    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:17:14.117369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:14.117338    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:14.534421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:14.530323    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:17:15.045145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.045102    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"]
Nov 03 13:17:15.049222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.049188    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.060523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.059839    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 13:17:15.060523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.060413    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l4zxf\""
Nov 03 13:17:15.060523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.060495    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 13:17:15.060717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.060626    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 13:17:15.062831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.062553    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"]
Nov 03 13:17:15.140832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.140763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:15.171878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.171834    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.172066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.171902    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.172066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.171934    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.172066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.171965    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmnzx\" (UniqueName: \"kubernetes.io/projected/3f80149c-8e25-4f79-b9ea-f06645646027-kube-api-access-rmnzx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.172066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.172010    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.172066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.172051    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f80149c-8e25-4f79-b9ea-f06645646027-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.272551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.272522    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f80149c-8e25-4f79-b9ea-f06645646027-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.272745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.272568    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.272745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.272607    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.272745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.272639    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.272745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.272669    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnzx\" (UniqueName: \"kubernetes.io/projected/3f80149c-8e25-4f79-b9ea-f06645646027-kube-api-access-rmnzx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.272745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.272715    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.273039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.273012    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.273252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.273197    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.273374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.273104    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.274983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.274954    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f80149c-8e25-4f79-b9ea-f06645646027-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.275627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.275607    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f80149c-8e25-4f79-b9ea-f06645646027-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.282382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.282360    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnzx\" (UniqueName: \"kubernetes.io/projected/3f80149c-8e25-4f79-b9ea-f06645646027-kube-api-access-rmnzx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7hvfd\" (UID: \"3f80149c-8e25-4f79-b9ea-f06645646027\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.369060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.368985    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:15.641974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:15.641794    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"]
Nov 03 13:17:15.645336 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:17:15.645308    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f80149c_8e25_4f79_b9ea_f06645646027.slice/crio-ec109e76d6244c6ca84058d321be33daeacbf3f712d9b3de76fdad52f189eff3 WatchSource:0}: Error finding container ec109e76d6244c6ca84058d321be33daeacbf3f712d9b3de76fdad52f189eff3: Status 404 returned error can't find the container with id ec109e76d6244c6ca84058d321be33daeacbf3f712d9b3de76fdad52f189eff3
Nov 03 13:17:16.224904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.224865    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd" event={"ID":"3f80149c-8e25-4f79-b9ea-f06645646027","Type":"ContainerStarted","Data":"ec109e76d6244c6ca84058d321be33daeacbf3f712d9b3de76fdad52f189eff3"}
Nov 03 13:17:16.606469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.606392    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"]
Nov 03 13:17:16.610191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.610166    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.625998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.625954    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 13:17:16.626138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.626048    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"]
Nov 03 13:17:16.686108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.686015    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.686108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.686060    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.686359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.686269    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4df66573-5b54-424a-be1f-f1e378e57ff5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.686359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.686327    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnsx6\" (UniqueName: \"kubernetes.io/projected/4df66573-5b54-424a-be1f-f1e378e57ff5-kube-api-access-lnsx6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.686429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.686357    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.686429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.686408    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.709672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.709642    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:16.787258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.787199    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.787451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.787289    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.787451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.787319    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.787451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.787338    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.787451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.787401    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4df66573-5b54-424a-be1f-f1e378e57ff5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.787451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.787443    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnsx6\" (UniqueName: \"kubernetes.io/projected/4df66573-5b54-424a-be1f-f1e378e57ff5-kube-api-access-lnsx6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.787670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.787622    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.787998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.787750    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.787998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.787960    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.789989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.789969    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4df66573-5b54-424a-be1f-f1e378e57ff5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.790177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.790156    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4df66573-5b54-424a-be1f-f1e378e57ff5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.796147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.796125    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnsx6\" (UniqueName: \"kubernetes.io/projected/4df66573-5b54-424a-be1f-f1e378e57ff5-kube-api-access-lnsx6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dgl5b\" (UID: \"4df66573-5b54-424a-be1f-f1e378e57ff5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:16.842263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.842232    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:16.927785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:16.927552    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:17.208425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:17.208401    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"]
Nov 03 13:17:17.210450 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:17:17.210424    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df66573_5b54_424a_be1f_f1e378e57ff5.slice/crio-1b3d0b01e14c26a8ed1949c0c52c2094959cfc7229ff9d5c33cff000ed06d033 WatchSource:0}: Error finding container 1b3d0b01e14c26a8ed1949c0c52c2094959cfc7229ff9d5c33cff000ed06d033: Status 404 returned error can't find the container with id 1b3d0b01e14c26a8ed1949c0c52c2094959cfc7229ff9d5c33cff000ed06d033
Nov 03 13:17:17.231467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:17.231439    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b" event={"ID":"4df66573-5b54-424a-be1f-f1e378e57ff5","Type":"ContainerStarted","Data":"1b3d0b01e14c26a8ed1949c0c52c2094959cfc7229ff9d5c33cff000ed06d033"}
Nov 03 13:17:17.234449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:17.233696    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd" event={"ID":"3f80149c-8e25-4f79-b9ea-f06645646027","Type":"ContainerStarted","Data":"87eeb3928ea9f14f7d8bfce16815617648704ba096d09c862a9fc6c3ef9ed47d"}
Nov 03 13:17:17.254775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:17.254730    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd" podStartSLOduration=0.866798329 podStartE2EDuration="2.254716036s" totalImagesPullingTime="1.387917707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:17.253666845 +0000 UTC m=+478.331431872" watchObservedRunningTime="2026-11-03 13:17:17.254716036 +0000 UTC m=+478.332481088"
Nov 03 13:17:17.258682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:17.258660    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:17:17.369933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:17.369904    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:18.241625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:18.241572    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b" event={"ID":"4df66573-5b54-424a-be1f-f1e378e57ff5","Type":"ContainerStarted","Data":"43e132efcd6cf0917958dd09d13ad77ea6193c084270cb7c384034c09b8bbec1"}
Nov 03 13:17:18.266002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:18.265940    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b" podStartSLOduration=1.898459218 podStartE2EDuration="2.265921843s" totalImagesPullingTime="367.462625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:18.265260479 +0000 UTC m=+479.343025506" watchObservedRunningTime="2026-11-03 13:17:18.265921843 +0000 UTC m=+479.343686870"
Nov 03 13:17:18.305387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:18.305357    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:18.928200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:18.928171    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:19.051948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:19.051917    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:19.479111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:19.479065    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:19.939468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:19.939386    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:20.859856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:20.859828    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:17:21.548695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.548666    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"]
Nov 03 13:17:21.552206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.552187    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.562038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.562017    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 13:17:21.562291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.562272    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"]
Nov 03 13:17:21.732259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.732227    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.732415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.732315    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8m5g\" (UniqueName: \"kubernetes.io/projected/5181ded3-5e13-47a1-98ab-64f7df8f8c64-kube-api-access-w8m5g\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.732415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.732376    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5181ded3-5e13-47a1-98ab-64f7df8f8c64-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.732415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.732409    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.732522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.732469    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.732522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.732483    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.833562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.833479    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.833562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.833515    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.833805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.833563    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.833805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.833599    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8m5g\" (UniqueName: \"kubernetes.io/projected/5181ded3-5e13-47a1-98ab-64f7df8f8c64-kube-api-access-w8m5g\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.833805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.833624    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5181ded3-5e13-47a1-98ab-64f7df8f8c64-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.833805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.833653    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.834096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.834045    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.834190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.834166    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.834328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.834200    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.836003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.835973    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5181ded3-5e13-47a1-98ab-64f7df8f8c64-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.836233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.836203    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5181ded3-5e13-47a1-98ab-64f7df8f8c64-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.842164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.842142    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8m5g\" (UniqueName: \"kubernetes.io/projected/5181ded3-5e13-47a1-98ab-64f7df8f8c64-kube-api-access-w8m5g\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh\" (UID: \"5181ded3-5e13-47a1-98ab-64f7df8f8c64\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.867154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.867132    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:21.896623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:21.896597    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:22.151405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:22.151375    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"]
Nov 03 13:17:22.153174 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:17:22.153145    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5181ded3_5e13_47a1_98ab_64f7df8f8c64.slice/crio-0b92d93423dbf1e4578f2a1cedcd976bb486c68cbd0977f26402ea51da2f3844 WatchSource:0}: Error finding container 0b92d93423dbf1e4578f2a1cedcd976bb486c68cbd0977f26402ea51da2f3844: Status 404 returned error can't find the container with id 0b92d93423dbf1e4578f2a1cedcd976bb486c68cbd0977f26402ea51da2f3844
Nov 03 13:17:22.275157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:22.275112    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh" event={"ID":"5181ded3-5e13-47a1-98ab-64f7df8f8c64","Type":"ContainerStarted","Data":"0b92d93423dbf1e4578f2a1cedcd976bb486c68cbd0977f26402ea51da2f3844"}
Nov 03 13:17:23.281726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.281692    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh" event={"ID":"5181ded3-5e13-47a1-98ab-64f7df8f8c64","Type":"ContainerStarted","Data":"1f66c22fdae6a2883855f2fdc76ee422be7f87cc2e226933fa2765d7ad49e688"}
Nov 03 13:17:23.298060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.298026    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"]
Nov 03 13:17:23.302067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.302044    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.311951 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.310109    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 13:17:23.313051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.312813    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh" podStartSLOduration=1.972100659 podStartE2EDuration="2.312797316s" totalImagesPullingTime="340.696657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:23.308743002 +0000 UTC m=+484.386508028" watchObservedRunningTime="2026-11-03 13:17:23.312797316 +0000 UTC m=+484.390562343"
Nov 03 13:17:23.316373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.316323    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"]
Nov 03 13:17:23.449741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.449699    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q55\" (UniqueName: \"kubernetes.io/projected/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-kube-api-access-l4q55\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.449741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.449746    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.450000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.449843    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.450000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.449890    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.450000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.449944    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.450179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.450039    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.502849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.502816    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:23.550725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.550634    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q55\" (UniqueName: \"kubernetes.io/projected/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-kube-api-access-l4q55\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.550725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.550681    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.550725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.550709    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.551014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.550737    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.551014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.550768    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.551014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.550787    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.551288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.551265    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.551382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.551292    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.551382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.551333    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.553037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.553016    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.553442 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.553419    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.570800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.570769    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q55\" (UniqueName: \"kubernetes.io/projected/0ee6c876-9bf8-4b10-89f8-60da53ccdfc1-kube-api-access-l4q55\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc\" (UID: \"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.618937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.618914    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:23.868285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.868179    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:23.898059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:23.898037    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"]
Nov 03 13:17:23.899872 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:17:23.899847    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ee6c876_9bf8_4b10_89f8_60da53ccdfc1.slice/crio-e2edeac5042660121f0c0a5b3015d6056d92fbea4a09081d54411c1aadb5183d WatchSource:0}: Error finding container e2edeac5042660121f0c0a5b3015d6056d92fbea4a09081d54411c1aadb5183d: Status 404 returned error can't find the container with id e2edeac5042660121f0c0a5b3015d6056d92fbea4a09081d54411c1aadb5183d
Nov 03 13:17:24.290538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:24.290509    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" event={"ID":"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1","Type":"ContainerStarted","Data":"e2edeac5042660121f0c0a5b3015d6056d92fbea4a09081d54411c1aadb5183d"}
Nov 03 13:17:24.735597 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:17:24.735564    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:17:24.967643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:24.967608    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:17:25.297409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:25.297377    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" event={"ID":"0ee6c876-9bf8-4b10-89f8-60da53ccdfc1","Type":"ContainerStarted","Data":"b2340eaa6f9b9e133e3bb3b2c9994244105691f8bebd622076b574b49268bd17"}
Nov 03 13:17:25.318575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:25.318532    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podStartSLOduration=1.987182073 podStartE2EDuration="2.318519045s" totalImagesPullingTime="331.336972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:25.316824088 +0000 UTC m=+486.394589134" watchObservedRunningTime="2026-11-03 13:17:25.318519045 +0000 UTC m=+486.396284105"
Nov 03 13:17:25.369734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:25.369708    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:25.382340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:25.382317    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:25.619580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:25.619546    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:26.325588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:26.325558    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7hvfd"
Nov 03 13:17:26.928282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:26.928249    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:26.941101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:26.941075    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:27.329409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:27.329378    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dgl5b"
Nov 03 13:17:31.867980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:31.867941    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:31.880531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:31.880502    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:32.349575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:32.349545    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh"
Nov 03 13:17:33.619720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:33.619691    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:33.632098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:33.632075    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:34.359484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:34.359448    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:17:39.023394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.023360    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"]
Nov 03 13:17:39.026425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.026403    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.051040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.051016    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"]
Nov 03 13:17:39.103131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.103103    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.103287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.103144    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62992ac-fb2f-447d-b8e4-83328832a8ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.103287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.103166    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.103368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.103307    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.103368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.103337    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.103462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.103444    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.103500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.103474    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k4zt\" (UniqueName: \"kubernetes.io/projected/d62992ac-fb2f-447d-b8e4-83328832a8ea-kube-api-access-4k4zt\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.103533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.103523    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.103566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.103540    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.204531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.204502    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62992ac-fb2f-447d-b8e4-83328832a8ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.204669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.204535    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.204669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.204563    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.204669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.204580    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.204669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.204627    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.204669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.204655    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k4zt\" (UniqueName: \"kubernetes.io/projected/d62992ac-fb2f-447d-b8e4-83328832a8ea-kube-api-access-4k4zt\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.204907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.204673    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.204907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.204689    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.205005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.204967    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.205062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.205044    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.205269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.205176    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.205269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.205244    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.205441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.205268    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.205441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.205359    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62992ac-fb2f-447d-b8e4-83328832a8ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.207010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.206982    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.207155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.207132    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.216368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.216345    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62992ac-fb2f-447d-b8e4-83328832a8ea-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.216612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.216597    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k4zt\" (UniqueName: \"kubernetes.io/projected/d62992ac-fb2f-447d-b8e4-83328832a8ea-kube-api-access-4k4zt\") pod \"maas-default-gateway-openshift-default-84587756dc-5v6m6\" (UID: \"d62992ac-fb2f-447d-b8e4-83328832a8ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.350123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.350052    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:39.631943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.631917    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"]
Nov 03 13:17:39.634012 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:17:39.633976    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62992ac_fb2f_447d_b8e4_83328832a8ea.slice/crio-f3cccbbd3bf9554c7dc2a0f524806f139e63b240443f22467f73baff2b6182a5 WatchSource:0}: Error finding container f3cccbbd3bf9554c7dc2a0f524806f139e63b240443f22467f73baff2b6182a5: Status 404 returned error can't find the container with id f3cccbbd3bf9554c7dc2a0f524806f139e63b240443f22467f73baff2b6182a5
Nov 03 13:17:39.636457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.636427    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:17:39.636532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.636492    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:17:39.636532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:39.636519    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:17:40.376762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:40.376725    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" event={"ID":"d62992ac-fb2f-447d-b8e4-83328832a8ea","Type":"ContainerStarted","Data":"c4a99119cfec982d100b31eff3faebc81bbee1e030e968a34cae1d50b9970781"}
Nov 03 13:17:40.377146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:40.376786    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" event={"ID":"d62992ac-fb2f-447d-b8e4-83328832a8ea","Type":"ContainerStarted","Data":"f3cccbbd3bf9554c7dc2a0f524806f139e63b240443f22467f73baff2b6182a5"}
Nov 03 13:17:40.399895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:40.399851    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podStartSLOduration=1.39983718 podStartE2EDuration="1.39983718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:40.39836125 +0000 UTC m=+501.476126302" watchObservedRunningTime="2026-11-03 13:17:40.39983718 +0000 UTC m=+501.477602206"
Nov 03 13:17:41.350576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:41.350508    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:41.351617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:41.351584    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 13:17:41.351743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:41.351651    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 13:17:42.351103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:42.351056    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 13:17:42.351617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:42.351126    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 13:17:43.351307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:43.351272    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 13:17:43.351803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:43.351326    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 13:17:44.047776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:44.047742    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:17:44.350927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:44.350852    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 13:17:44.351070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:44.350945    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 13:17:45.350728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:45.350683    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 13:17:45.351258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:45.350770    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 13:17:46.351142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:46.351105    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 13:17:46.351701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:46.351674    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 13:17:47.642036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:47.641999    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:47.895467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:47.895382    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:48.352181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:48.352143    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:17:48.352404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:48.352204    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:17:49.153163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:49.153121    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:49.295157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:49.295130    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:49.393350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:49.393314    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:17:49.393521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:49.393364    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:17:49.841376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:49.841350    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:49.841642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:49.841625    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:49.842347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:49.842330    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6"
Nov 03 13:17:54.056179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.056143    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"]
Nov 03 13:17:54.063075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.063051    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.072954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.072929    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"]
Nov 03 13:17:54.086192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.086164    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"]
Nov 03 13:17:54.090911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.090889    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.097486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.097463    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"]
Nov 03 13:17:54.168066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.168039    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.168224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.168074    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.168224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.168155    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.168316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.168242    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.168316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.168303    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.168383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.168349    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.168424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.168392    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b627\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-kube-api-access-9b627\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.168424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.168419    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.168525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.168499    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.269573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269542    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.269729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269578    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.269729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269603    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhww\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-kube-api-access-wvhww\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.269729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269620    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.269729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269643    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.269944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269741    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.269944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269766    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.269944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269863    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.269944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269888    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.269944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269908    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b627\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-kube-api-access-9b627\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269986    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.270161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.269991    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270002    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270043    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.270161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270084    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270107    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.270161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270124    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270191    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270224    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270249    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270271    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270512    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.270699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.270681    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.272374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.272351    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.272637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.272621    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.277860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.277834    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b627\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-kube-api-access-9b627\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.277955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.277878    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xz4c4\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.371028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.370951    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhww\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-kube-api-access-wvhww\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.370994    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371020    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371044    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371095    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371151    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371185    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371251    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371332    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371450    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371530    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371632    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371704    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.371947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.371924    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.373454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.373431    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.373454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.373448    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.377764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.377739    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:54.379281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.379256    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhww\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-kube-api-access-wvhww\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.379403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.379320    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-ttvvr\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.406568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.406543    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:54.661291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.661267    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"]
Nov 03 13:17:54.662974 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:17:54.662947    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1cd1f4_1249_4347_b5bf_f75d94de573e.slice/crio-7b2ea0559f51dea24772874c5d6cf2a0294bcc7b0db434b5aded9a8bca33cf0a WatchSource:0}: Error finding container 7b2ea0559f51dea24772874c5d6cf2a0294bcc7b0db434b5aded9a8bca33cf0a: Status 404 returned error can't find the container with id 7b2ea0559f51dea24772874c5d6cf2a0294bcc7b0db434b5aded9a8bca33cf0a
Nov 03 13:17:54.664953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.664925    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:17:54.665016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.664986    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:17:54.665052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.665018    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:17:54.693744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.693722    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"]
Nov 03 13:17:54.695159 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:17:54.695137    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29ba0ec_171e_44ef_96e0_e8fbd2a73e27.slice/crio-b3b4c43ae2cc3baef1d582d37434a42ab7b9a22d92d1adcc521d1a2b75624b3b WatchSource:0}: Error finding container b3b4c43ae2cc3baef1d582d37434a42ab7b9a22d92d1adcc521d1a2b75624b3b: Status 404 returned error can't find the container with id b3b4c43ae2cc3baef1d582d37434a42ab7b9a22d92d1adcc521d1a2b75624b3b
Nov 03 13:17:54.697288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.697257    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:17:54.697381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.697330    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:17:54.697449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:54.697378    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:17:54.731286 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:17:54.731254    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:17:55.470303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:55.470256    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" event={"ID":"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27","Type":"ContainerStarted","Data":"7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce"}
Nov 03 13:17:55.470821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:55.470388    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" event={"ID":"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27","Type":"ContainerStarted","Data":"b3b4c43ae2cc3baef1d582d37434a42ab7b9a22d92d1adcc521d1a2b75624b3b"}
Nov 03 13:17:55.473099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:55.473064    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" event={"ID":"6f1cd1f4-1249-4347-b5bf-f75d94de573e","Type":"ContainerStarted","Data":"11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc"}
Nov 03 13:17:55.473271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:55.473122    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" event={"ID":"6f1cd1f4-1249-4347-b5bf-f75d94de573e","Type":"ContainerStarted","Data":"7b2ea0559f51dea24772874c5d6cf2a0294bcc7b0db434b5aded9a8bca33cf0a"}
Nov 03 13:17:55.493722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:55.493677    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" podStartSLOduration=1.493664249 podStartE2EDuration="1.493664249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:55.492156429 +0000 UTC m=+516.569921455" watchObservedRunningTime="2026-11-03 13:17:55.493664249 +0000 UTC m=+516.571429329"
Nov 03 13:17:55.517629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:55.517584    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" podStartSLOduration=1.517569689 podStartE2EDuration="1.517569689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:55.515560767 +0000 UTC m=+516.593325790" watchObservedRunningTime="2026-11-03 13:17:55.517569689 +0000 UTC m=+516.595334715"
Nov 03 13:17:56.378911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:56.378427    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:17:56.379795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:56.379773    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xz4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 13:17:56.379852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:56.379829    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 13:17:56.407652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:56.407624    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:17:56.408557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:56.408537    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ttvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 13:17:56.408630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:56.408599    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 13:17:57.378895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:57.378853    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xz4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 13:17:57.379475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:57.378900    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 13:17:57.407182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:57.407151    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ttvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 13:17:57.407343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:57.407206    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 13:17:58.378918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:58.378877    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xz4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 13:17:58.379339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:58.378936    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 13:17:58.407855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:58.407828    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ttvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 13:17:58.408006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:58.407881    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 13:17:59.379119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:59.378724    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xz4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 13:17:59.379119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:59.378780    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 13:17:59.407740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:59.407709    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ttvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 13:17:59.407882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:17:59.407752    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 13:18:00.378581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:00.378540    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xz4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 13:18:00.378767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:00.378595    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 13:18:00.407171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:00.407141    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ttvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 13:18:00.407545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:00.407180    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 13:18:01.379044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:01.379010    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xz4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 13:18:01.379290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:01.379067    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 13:18:01.407364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:01.407321    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ttvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 13:18:01.407856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:01.407374    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 13:18:02.967354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:02.967320    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:18:02.968574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:02.968548    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:18:02.969387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:02.969365    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:18:03.008857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:03.008831    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:18:03.009333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:03.009313    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:18:03.009966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:03.009939    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:18:15.698692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:15.698650    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:16.054066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.054033    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"]
Nov 03 13:18:16.057845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.057825    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.067785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.067760    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e440bc-openshift-default-dockercfg-84hml\""
Nov 03 13:18:16.070793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.070769    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"]
Nov 03 13:18:16.187852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.187816    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.187852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.187856    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.188078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.187941    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqsvs\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-kube-api-access-wqsvs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.188078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.187983    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.188249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.188204    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.188332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.188280    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6ec136-db56-450a-81db-bdad382f9884-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.188395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.188378    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.188448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.188426    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6ec136-db56-450a-81db-bdad382f9884-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.188509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.188492    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.288987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.288955    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqsvs\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-kube-api-access-wqsvs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.288994    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289026    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289149    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6ec136-db56-450a-81db-bdad382f9884-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289201    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289252    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6ec136-db56-450a-81db-bdad382f9884-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289280    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289379    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289421    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289510    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289531    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289614    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.289858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.289825    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.290498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.290472    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6ec136-db56-450a-81db-bdad382f9884-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.291909 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.291889    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.292334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.292316    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6ec136-db56-450a-81db-bdad382f9884-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.303454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.303430    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.303605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.303583    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqsvs\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-kube-api-access-wqsvs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.373419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.373346    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:16.691554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.691524    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"]
Nov 03 13:18:16.693536 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:16.693505    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6ec136_db56_450a_81db_bdad382f9884.slice/crio-04cc8e525a313b5eba9ab8cbdee34fb2e90241b31d953e8b0b7f9f7183e6f2b8 WatchSource:0}: Error finding container 04cc8e525a313b5eba9ab8cbdee34fb2e90241b31d953e8b0b7f9f7183e6f2b8: Status 404 returned error can't find the container with id 04cc8e525a313b5eba9ab8cbdee34fb2e90241b31d953e8b0b7f9f7183e6f2b8
Nov 03 13:18:16.695625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.695596    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:18:16.695689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.695659    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:18:16.695725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.695689    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:18:16.985656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:16.985625    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:17.585651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:17.585596    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" event={"ID":"9a6ec136-db56-450a-81db-bdad382f9884","Type":"ContainerStarted","Data":"25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605"}
Nov 03 13:18:17.585848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:17.585672    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" event={"ID":"9a6ec136-db56-450a-81db-bdad382f9884","Type":"ContainerStarted","Data":"04cc8e525a313b5eba9ab8cbdee34fb2e90241b31d953e8b0b7f9f7183e6f2b8"}
Nov 03 13:18:17.611882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:17.611813    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podStartSLOduration=1.611797724 podStartE2EDuration="1.611797724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:17.608953883 +0000 UTC m=+538.686718909" watchObservedRunningTime="2026-11-03 13:18:17.611797724 +0000 UTC m=+538.689562750"
Nov 03 13:18:18.374322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:18.374290    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:18.379700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:18.379672    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:18.591001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:18.590975    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:18.591949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:18.591930    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:19.890978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:19.890944    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:20.066413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:20.066383    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:20.227322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:20.227288    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:20.401505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:20.401474    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:21.123933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.123906    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:21.167584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.167553    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"]
Nov 03 13:18:21.172198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.172174    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:21.173868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.173845    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"]
Nov 03 13:18:21.177321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.177305    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:21.180698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.180677    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"]
Nov 03 13:18:21.182782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.182756    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e440bc\""
Nov 03 13:18:21.182915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.182889    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e440bc-dockercfg-wjnmw\""
Nov 03 13:18:21.190372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.190348    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"]
Nov 03 13:18:21.236521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.236484    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttlg\" (UniqueName: \"kubernetes.io/projected/d9eda720-9a95-420a-bc9b-22d9c080177d-kube-api-access-7ttlg\") pod \"payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g\" (UID: \"d9eda720-9a95-420a-bc9b-22d9c080177d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:21.236669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.236549    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eda720-9a95-420a-bc9b-22d9c080177d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g\" (UID: \"d9eda720-9a95-420a-bc9b-22d9c080177d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:21.236669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.236632    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m\" (UID: \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:21.236669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.236658    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92cn2\" (UniqueName: \"kubernetes.io/projected/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-kube-api-access-92cn2\") pod \"payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m\" (UID: \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:21.269746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.269716    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:21.338132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.338100    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m\" (UID: \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:21.338132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.338142    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92cn2\" (UniqueName: \"kubernetes.io/projected/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-kube-api-access-92cn2\") pod \"payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m\" (UID: \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:21.338413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.338316    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttlg\" (UniqueName: \"kubernetes.io/projected/d9eda720-9a95-420a-bc9b-22d9c080177d-kube-api-access-7ttlg\") pod \"payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g\" (UID: \"d9eda720-9a95-420a-bc9b-22d9c080177d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:21.338413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.338370    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eda720-9a95-420a-bc9b-22d9c080177d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g\" (UID: \"d9eda720-9a95-420a-bc9b-22d9c080177d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:21.339100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.339068    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m\" (UID: \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:21.339220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.339193    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eda720-9a95-420a-bc9b-22d9c080177d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g\" (UID: \"d9eda720-9a95-420a-bc9b-22d9c080177d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:21.350288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.350265    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92cn2\" (UniqueName: \"kubernetes.io/projected/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-kube-api-access-92cn2\") pod \"payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m\" (UID: \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:21.350394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.350272    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttlg\" (UniqueName: \"kubernetes.io/projected/d9eda720-9a95-420a-bc9b-22d9c080177d-kube-api-access-7ttlg\") pod \"payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g\" (UID: \"d9eda720-9a95-420a-bc9b-22d9c080177d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:21.408034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.407957    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:21.500659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.500634    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:21.502711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.502683    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:21.781051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.781023    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"]
Nov 03 13:18:21.782841 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:21.782809    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d6d9b0_3097_4b04_b6f9_ebc9fa6c2a80.slice/crio-d24da4ca52c058adf325438134afbed6b056d12dc128d94c2fb9b8405212cc65 WatchSource:0}: Error finding container d24da4ca52c058adf325438134afbed6b056d12dc128d94c2fb9b8405212cc65: Status 404 returned error can't find the container with id d24da4ca52c058adf325438134afbed6b056d12dc128d94c2fb9b8405212cc65
Nov 03 13:18:21.783118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.783099    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"]
Nov 03 13:18:21.783733 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:21.783706    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9eda720_9a95_420a_bc9b_22d9c080177d.slice/crio-d59056afd5e9b78b998ec74c851e8db0b8ef17d55e528cc510766e53bd5f77ef WatchSource:0}: Error finding container d59056afd5e9b78b998ec74c851e8db0b8ef17d55e528cc510766e53bd5f77ef: Status 404 returned error can't find the container with id d59056afd5e9b78b998ec74c851e8db0b8ef17d55e528cc510766e53bd5f77ef
Nov 03 13:18:21.944521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.944492    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:21.962242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.962204    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"]
Nov 03 13:18:21.967082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.967065    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:21.974761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.974741    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c381a7\""
Nov 03 13:18:21.974915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.974894    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c381a7-dockercfg-hmf5s\""
Nov 03 13:18:21.977763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:21.977744    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"]
Nov 03 13:18:22.046087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.046024    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts\" (UID: \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:22.046087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.046082    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drw4\" (UniqueName: \"kubernetes.io/projected/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-kube-api-access-2drw4\") pod \"payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts\" (UID: \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:22.095436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.095393    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:22.147718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.147688    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts\" (UID: \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:22.148200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.147779    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drw4\" (UniqueName: \"kubernetes.io/projected/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-kube-api-access-2drw4\") pod \"payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts\" (UID: \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:22.148531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.148512    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts\" (UID: \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:22.157243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.157202    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drw4\" (UniqueName: \"kubernetes.io/projected/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-kube-api-access-2drw4\") pod \"payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts\" (UID: \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:22.230724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.230693    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:22.281297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.281265    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:22.433952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.433898    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:22.489742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.489709    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"]
Nov 03 13:18:22.493521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.493498    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:22.511549 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.511526    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"]
Nov 03 13:18:22.539628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.539594    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a4ab8e\""
Nov 03 13:18:22.571993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.571960    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:22.596073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.595984    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a4ab8e-dockercfg-mt4ql\""
Nov 03 13:18:22.598076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.598052    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"]
Nov 03 13:18:22.615895 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:22.615864    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bd0a5a_58c6_4bd6_b731_c2c7743c4b65.slice/crio-528c934a2afc95afcb9076aa154833113ed8b4c2d46f3085a9a99b2cd7366580 WatchSource:0}: Error finding container 528c934a2afc95afcb9076aa154833113ed8b4c2d46f3085a9a99b2cd7366580: Status 404 returned error can't find the container with id 528c934a2afc95afcb9076aa154833113ed8b4c2d46f3085a9a99b2cd7366580
Nov 03 13:18:22.624198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.624167    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g" event={"ID":"d9eda720-9a95-420a-bc9b-22d9c080177d","Type":"ContainerStarted","Data":"d59056afd5e9b78b998ec74c851e8db0b8ef17d55e528cc510766e53bd5f77ef"}
Nov 03 13:18:22.626427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.626404    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m" event={"ID":"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80","Type":"ContainerStarted","Data":"d24da4ca52c058adf325438134afbed6b056d12dc128d94c2fb9b8405212cc65"}
Nov 03 13:18:22.677137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.677098    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbwd\" (UniqueName: \"kubernetes.io/projected/a2aab2b3-fa9c-472e-b052-796540fc6e20-kube-api-access-hmbwd\") pod \"payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p\" (UID: \"a2aab2b3-fa9c-472e-b052-796540fc6e20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:22.677369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.677280    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2aab2b3-fa9c-472e-b052-796540fc6e20-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p\" (UID: \"a2aab2b3-fa9c-472e-b052-796540fc6e20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:22.778537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.778499    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbwd\" (UniqueName: \"kubernetes.io/projected/a2aab2b3-fa9c-472e-b052-796540fc6e20-kube-api-access-hmbwd\") pod \"payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p\" (UID: \"a2aab2b3-fa9c-472e-b052-796540fc6e20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:22.778718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.778672    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2aab2b3-fa9c-472e-b052-796540fc6e20-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p\" (UID: \"a2aab2b3-fa9c-472e-b052-796540fc6e20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:22.779601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.779574    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2aab2b3-fa9c-472e-b052-796540fc6e20-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p\" (UID: \"a2aab2b3-fa9c-472e-b052-796540fc6e20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:22.792299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.792277    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbwd\" (UniqueName: \"kubernetes.io/projected/a2aab2b3-fa9c-472e-b052-796540fc6e20-kube-api-access-hmbwd\") pod \"payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p\" (UID: \"a2aab2b3-fa9c-472e-b052-796540fc6e20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:22.816154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.816131    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:22.884971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:22.884888    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:23.110928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:23.110578    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"]
Nov 03 13:18:23.112495 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:23.112466    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2aab2b3_fa9c_472e_b052_796540fc6e20.slice/crio-5c5a0d8220d11f4c2051a7b7526f3babefc4b695e495aac441c437460061bc5c WatchSource:0}: Error finding container 5c5a0d8220d11f4c2051a7b7526f3babefc4b695e495aac441c437460061bc5c: Status 404 returned error can't find the container with id 5c5a0d8220d11f4c2051a7b7526f3babefc4b695e495aac441c437460061bc5c
Nov 03 13:18:23.379561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:23.378921    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:18:23.512343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:23.511781    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:23.637329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:23.636771    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts" event={"ID":"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65","Type":"ContainerStarted","Data":"528c934a2afc95afcb9076aa154833113ed8b4c2d46f3085a9a99b2cd7366580"}
Nov 03 13:18:23.639574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:23.639543    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p" event={"ID":"a2aab2b3-fa9c-472e-b052-796540fc6e20","Type":"ContainerStarted","Data":"5c5a0d8220d11f4c2051a7b7526f3babefc4b695e495aac441c437460061bc5c"}
Nov 03 13:18:23.648931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:23.648735    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:24.215038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:24.214971    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:24.362277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:24.359973    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:24.492682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:24.492549    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:24.779184 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:18:24.779040    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:18:24.884182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:24.884119    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:25.497557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:25.496287    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:25.663359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:25.663327    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m" event={"ID":"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80","Type":"ContainerStarted","Data":"7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a"}
Nov 03 13:18:25.663869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:25.663851    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:25.687276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:25.687228    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m" podStartSLOduration=1.1766806380000001 podStartE2EDuration="4.687195553s" totalImagesPullingTime="3.510514915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:25.683783359 +0000 UTC m=+546.761548364" watchObservedRunningTime="2026-11-03 13:18:25.687195553 +0000 UTC m=+546.764960614"
Nov 03 13:18:26.109057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.108993    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:26.673746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.673656    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p" event={"ID":"a2aab2b3-fa9c-472e-b052-796540fc6e20","Type":"ContainerStarted","Data":"4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f"}
Nov 03 13:18:26.674239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.673778    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:26.676315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.675963    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g" event={"ID":"d9eda720-9a95-420a-bc9b-22d9c080177d","Type":"ContainerStarted","Data":"fe1dd7eaf7f2deed185d8207164a6ab77a7d1db544bb472a6e001aff089e1d09"}
Nov 03 13:18:26.678091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.676873    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:26.682735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.682708    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts" event={"ID":"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65","Type":"ContainerStarted","Data":"c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7"}
Nov 03 13:18:26.682858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.682759    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:26.714238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.714048    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p" podStartSLOduration=2.278048501 podStartE2EDuration="4.714032583s" totalImagesPullingTime="2.435984082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:26.712624903 +0000 UTC m=+547.790389928" watchObservedRunningTime="2026-11-03 13:18:26.714032583 +0000 UTC m=+547.791797609"
Nov 03 13:18:26.766233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.766162    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g" podStartSLOduration=2.061689308 podStartE2EDuration="5.766147169s" totalImagesPullingTime="3.704457861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:26.765048126 +0000 UTC m=+547.842813151" watchObservedRunningTime="2026-11-03 13:18:26.766147169 +0000 UTC m=+547.843912186"
Nov 03 13:18:26.796969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.796915    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts" podStartSLOduration=2.507707821 podStartE2EDuration="5.796899923s" totalImagesPullingTime="3.289192102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:26.795162354 +0000 UTC m=+547.872927394" watchObservedRunningTime="2026-11-03 13:18:26.796899923 +0000 UTC m=+547.874664951"
Nov 03 13:18:26.881542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:26.881512    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:27.606836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:27.606796    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:28.213781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:28.213706    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:28.793737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:28.793706    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:29.404543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:29.404508    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:29.959437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:29.959402    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:18:30.086758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:30.086665    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:30.722325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:30.721649    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:31.468399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:31.468351    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:32.176131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:32.176059    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:32.812829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:32.812709    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:33.512483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:33.512453    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:33.592504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:33.592467    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 13:18:33.592697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:33.592539    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 13:18:33.726077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:33.726040    5420 generic.go:356] "Generic (PLEG): container finished" podID="9a6ec136-db56-450a-81db-bdad382f9884" containerID="25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605" exitCode=0
Nov 03 13:18:33.726261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:33.726113    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" event={"ID":"9a6ec136-db56-450a-81db-bdad382f9884","Type":"ContainerDied","Data":"25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605"}
Nov 03 13:18:33.726588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:33.726574    5420 scope.go:122] "RemoveContainer" containerID="25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605"
Nov 03 13:18:33.727704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:33.727679    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:18:33.727753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:33.727741    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:18:33.727787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:33.727772    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:18:34.198860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:34.198771    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:34.748578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:34.748538    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" event={"ID":"9a6ec136-db56-450a-81db-bdad382f9884","Type":"ContainerStarted","Data":"4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497"}
Nov 03 13:18:34.789706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:34.789678    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:34.946512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:34.946478    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:35.083587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.083503    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:35.374425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.374343    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:35.374604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.374501    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 13:18:35.374604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.374563    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 13:18:35.418419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.418394    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:35.654180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.654101    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"]
Nov 03 13:18:35.658495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.658475    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.674427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.674404    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"]
Nov 03 13:18:35.678246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.678224    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:18:35.678355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.678313    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:18:35.678355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.678326    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:18:35.678355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.678311    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:18:35.715045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.715024    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:35.760464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.760434    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkntl\" (UniqueName: \"kubernetes.io/projected/bc6a2b36-49d6-42db-9f69-d51155955377-kube-api-access-hkntl\") pod \"maas-api-revoke-sub-bc7c2d03-hmgbt\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.760913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.760634    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7c2d03-hmgbt\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.760913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.760688    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7c2d03-hmgbt\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.863357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.862738    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7c2d03-hmgbt\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.863828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.863709    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7c2d03-hmgbt\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.864289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.864271    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkntl\" (UniqueName: \"kubernetes.io/projected/bc6a2b36-49d6-42db-9f69-d51155955377-kube-api-access-hkntl\") pod \"maas-api-revoke-sub-bc7c2d03-hmgbt\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.868917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.868894    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7c2d03-hmgbt\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.869515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.869495    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7c2d03-hmgbt\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.877971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.877944    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkntl\" (UniqueName: \"kubernetes.io/projected/bc6a2b36-49d6-42db-9f69-d51155955377-kube-api-access-hkntl\") pod \"maas-api-revoke-sub-bc7c2d03-hmgbt\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:35.895681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.895647    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:35.971852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:35.971826    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:36.267065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:36.267042    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"]
Nov 03 13:18:36.269196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:36.269168    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6a2b36_49d6_42db_9f69_d51155955377.slice/crio-63beb4458f5f4eafe5c316ecabbcb7af3788fe1b2b5aa180ab29776ba228c79e WatchSource:0}: Error finding container 63beb4458f5f4eafe5c316ecabbcb7af3788fe1b2b5aa180ab29776ba228c79e: Status 404 returned error can't find the container with id 63beb4458f5f4eafe5c316ecabbcb7af3788fe1b2b5aa180ab29776ba228c79e
Nov 03 13:18:36.374406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:36.374378    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 13:18:36.374560 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:36.374422    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 13:18:36.487971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:36.487939    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:18:36.560534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:36.560452    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:36.685697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:36.685671    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:18:36.760552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:36.760521    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt" event={"ID":"bc6a2b36-49d6-42db-9f69-d51155955377","Type":"ContainerStarted","Data":"63beb4458f5f4eafe5c316ecabbcb7af3788fe1b2b5aa180ab29776ba228c79e"}
Nov 03 13:18:37.314279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:37.314206    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:37.377828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:37.377791    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 13:18:37.378117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:37.378088    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 13:18:37.693528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:37.693445    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:18:37.694363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:37.694339    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:18:38.029696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:38.029604    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:38.387289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:38.386260    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 13:18:38.387289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:38.386333    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 13:18:38.723613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:38.722635    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:18:38.813236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:38.809839    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"]
Nov 03 13:18:38.990326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:38.965253    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:39.062240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.058804    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"]
Nov 03 13:18:39.062240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.058917    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.072586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.072558    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"openshift-service-ca.crt\""
Nov 03 13:18:39.072983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.072962    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"default-dockercfg-5nftw\""
Nov 03 13:18:39.088513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.088490    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"kube-root-ca.crt\""
Nov 03 13:18:39.118401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.108197    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"simulator-w2-e440bc-kserve-self-signed-certs\""
Nov 03 13:18:39.226471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.224799    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-dshm\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.226471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.224880    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-home\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.226471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.224928    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-model-cache\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.226471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.224959    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hh9m\" (UniqueName: \"kubernetes.io/projected/93865294-a4e8-40fc-a2e2-d61659934afa-kube-api-access-9hh9m\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.226471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.224989    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93865294-a4e8-40fc-a2e2-d61659934afa-tls-certs\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.226471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.225031    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-tmp-dir\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.246234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.246102    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:39.366732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.366679    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-dshm\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.367906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.367847    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-home\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.368115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.368093    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-home\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.387741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.387714    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-model-cache\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.388093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.376245    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 13:18:39.388450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.388258    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 13:18:39.388745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.388728    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-dshm\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.389037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.389019    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hh9m\" (UniqueName: \"kubernetes.io/projected/93865294-a4e8-40fc-a2e2-d61659934afa-kube-api-access-9hh9m\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.390315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.389781    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93865294-a4e8-40fc-a2e2-d61659934afa-tls-certs\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.402722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.402306    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-tmp-dir\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.403101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.395735    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93865294-a4e8-40fc-a2e2-d61659934afa-tls-certs\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.407343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.389726    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-model-cache\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.407582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.407564    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-tmp-dir\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.432333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.432309    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hh9m\" (UniqueName: \"kubernetes.io/projected/93865294-a4e8-40fc-a2e2-d61659934afa-kube-api-access-9hh9m\") pod \"simulator-w2-e440bc-kserve-7697659859-frndz\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") " pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.567473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.567394    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:39.931616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:39.931398    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:40.600495 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:40.596143    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93865294_a4e8_40fc_a2e2_d61659934afa.slice/crio-ceef284c80bc870980af76d212f0634ce1638544b41ceea868a13b6e48d77736 WatchSource:0}: Error finding container ceef284c80bc870980af76d212f0634ce1638544b41ceea868a13b6e48d77736: Status 404 returned error can't find the container with id ceef284c80bc870980af76d212f0634ce1638544b41ceea868a13b6e48d77736
Nov 03 13:18:40.626239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:40.613803    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 13:18:40.626239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:40.614100    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 13:18:40.643234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:40.641309    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:40.657236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:40.653835    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"]
Nov 03 13:18:40.804324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:40.780869    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:41.067575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.067432    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:41.222875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.222666    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:41.224065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.223201    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" event={"ID":"93865294-a4e8-40fc-a2e2-d61659934afa","Type":"ContainerStarted","Data":"2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf"}
Nov 03 13:18:41.224065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.223269    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" event={"ID":"93865294-a4e8-40fc-a2e2-d61659934afa","Type":"ContainerStarted","Data":"ceef284c80bc870980af76d212f0634ce1638544b41ceea868a13b6e48d77736"}
Nov 03 13:18:41.246579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.246528    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podStartSLOduration=3.246509493 podStartE2EDuration="3.246509493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:41.245370137 +0000 UTC m=+562.323135164" watchObservedRunningTime="2026-11-03 13:18:41.246509493 +0000 UTC m=+562.324274518"
Nov 03 13:18:41.379767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.379697    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:41.380363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.380338    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:41.381088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.381066    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:18:41.568778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.568725    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:41.703806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:41.703773    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:42.047152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.047073    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"]
Nov 03 13:18:42.072249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.066030    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"]
Nov 03 13:18:42.072249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.066143    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.077256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.074651    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"default-dockercfg-d8rbk\""
Nov 03 13:18:42.077256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.074729    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"openshift-service-ca.crt\""
Nov 03 13:18:42.077256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.074848    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"simulator-w4-c381a7-kserve-self-signed-certs\""
Nov 03 13:18:42.077256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.076656    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"kube-root-ca.crt\""
Nov 03 13:18:42.100195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.100171    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tls-certs\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.100344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.100203    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8bb\" (UniqueName: \"kubernetes.io/projected/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-kube-api-access-xr8bb\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.100344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.100327    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-dshm\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.100480 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.100360    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-home\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.100541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.100499    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-model-cache\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.100541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.100522    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tmp-dir\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.122247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.122179    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:42.202126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.202092    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-dshm\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.202297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.202142    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-home\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.202297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.202276    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-model-cache\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.202392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.202303    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tmp-dir\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.202392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.202384    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tls-certs\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.202474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.202419    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8bb\" (UniqueName: \"kubernetes.io/projected/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-kube-api-access-xr8bb\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.202826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.202797    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-model-cache\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.203099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.203076    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tmp-dir\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.203260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.203244    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-home\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.205022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.204946    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-dshm\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.205916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.205837    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tls-certs\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.211862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.211838    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8bb\" (UniqueName: \"kubernetes.io/projected/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-kube-api-access-xr8bb\") pod \"simulator-w4-c381a7-kserve-765c79788-fgssv\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") " pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.401338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.401263    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:42.476235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.475411    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:42.732391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:42.732334    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"]
Nov 03 13:18:43.180235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:43.170167    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:43.262078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:43.262042    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" event={"ID":"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a","Type":"ContainerStarted","Data":"19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0"}
Nov 03 13:18:43.262278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:43.262095    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" event={"ID":"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a","Type":"ContainerStarted","Data":"9d235635c287edc395f397a3f4e9d511cedf090ebabcfbe71d4a40369ffad4bb"}
Nov 03 13:18:43.280786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:43.280710    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podStartSLOduration=1.280690853 podStartE2EDuration="1.280690853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:43.278158659 +0000 UTC m=+564.355923687" watchObservedRunningTime="2026-11-03 13:18:43.280690853 +0000 UTC m=+564.358455880"
Nov 03 13:18:43.402144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:43.402110    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:43.801828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:43.801795    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:43.983080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:43.983045    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:18:44.280231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:44.279576    5420 generic.go:356] "Generic (PLEG): container finished" podID="bc6a2b36-49d6-42db-9f69-d51155955377" containerID="98a2a7f1bd2c8404ba7c57f7d00e44788802cf74c363cad72f015c5e7a088c69" exitCode=0
Nov 03 13:18:44.280405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:44.280307    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt" event={"ID":"bc6a2b36-49d6-42db-9f69-d51155955377","Type":"ContainerDied","Data":"98a2a7f1bd2c8404ba7c57f7d00e44788802cf74c363cad72f015c5e7a088c69"}
Nov 03 13:18:44.507703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:44.507670    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:45.422266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.422193    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:45.446676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.446654    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:45.539597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.539497    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6a2b36-49d6-42db-9f69-d51155955377-kube-api-access-hkntl\" (UniqueName: \"kubernetes.io/projected/bc6a2b36-49d6-42db-9f69-d51155955377-kube-api-access-hkntl\") pod \"bc6a2b36-49d6-42db-9f69-d51155955377\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") "
Nov 03 13:18:45.539597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.539569    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-openshift-ca\") pod \"bc6a2b36-49d6-42db-9f69-d51155955377\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") "
Nov 03 13:18:45.539790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.539644    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-xks-ca\") pod \"bc6a2b36-49d6-42db-9f69-d51155955377\" (UID: \"bc6a2b36-49d6-42db-9f69-d51155955377\") "
Nov 03 13:18:45.539959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.539931    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-openshift-ca" pod "bc6a2b36-49d6-42db-9f69-d51155955377" (UID: "bc6a2b36-49d6-42db-9f69-d51155955377"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:45.541830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.541799    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6a2b36-49d6-42db-9f69-d51155955377-kube-api-access-hkntl" pod "bc6a2b36-49d6-42db-9f69-d51155955377" (UID: "bc6a2b36-49d6-42db-9f69-d51155955377"). InnerVolumeSpecName "kube-api-access-hkntl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:45.541830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.541803    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-xks-ca" pod "bc6a2b36-49d6-42db-9f69-d51155955377" (UID: "bc6a2b36-49d6-42db-9f69-d51155955377"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:45.641220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.641178    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:45.641220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.641205    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkntl\" (UniqueName: \"kubernetes.io/projected/bc6a2b36-49d6-42db-9f69-d51155955377-kube-api-access-hkntl\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:45.641220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:45.641227    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6a2b36-49d6-42db-9f69-d51155955377-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:46.000797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.000760    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:46.292066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.291979    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"
Nov 03 13:18:46.292066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.291983    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt" event={"ID":"bc6a2b36-49d6-42db-9f69-d51155955377","Type":"ContainerDied","Data":"63beb4458f5f4eafe5c316ecabbcb7af3788fe1b2b5aa180ab29776ba228c79e"}
Nov 03 13:18:46.292066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.292041    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63beb4458f5f4eafe5c316ecabbcb7af3788fe1b2b5aa180ab29776ba228c79e"
Nov 03 13:18:46.623478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.623374    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:46.940944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.940863    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"]
Nov 03 13:18:46.941397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.941378    5420 cpu_manager.go:414] "containerMap: removing container" podUID="bc6a2b36-49d6-42db-9f69-d51155955377" containerName="revoke-keys"
Nov 03 13:18:46.941397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.941398    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6a2b36-49d6-42db-9f69-d51155955377" containerName="revoke-keys"
Nov 03 13:18:46.941543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.941475    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6a2b36-49d6-42db-9f69-d51155955377" containerName="revoke-keys"
Nov 03 13:18:46.944436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.944415    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:46.950127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.950102    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a4ab8e\"/\"kube-root-ca.crt\""
Nov 03 13:18:46.951439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.951413    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a4ab8e\"/\"simulator-w0-a4ab8e-kserve-self-signed-certs\""
Nov 03 13:18:46.952393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.952368    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a4ab8e\"/\"openshift-service-ca.crt\""
Nov 03 13:18:46.952570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.952403    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a4ab8e\"/\"default-dockercfg-89xvk\""
Nov 03 13:18:46.955730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:46.955710    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"]
Nov 03 13:18:47.044582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.044548    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:47.054875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.054834    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-home\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.054971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.054913    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpq6x\" (UniqueName: \"kubernetes.io/projected/fbbb9af9-1713-44f4-afcd-901667677352-kube-api-access-xpq6x\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.055031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.054971    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbb9af9-1713-44f4-afcd-901667677352-tls-certs\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.055109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.055088    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-tmp-dir\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.055152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.055121    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-model-cache\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.055152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.055141    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-dshm\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.155951 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.155913    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-home\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.156151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.155986    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpq6x\" (UniqueName: \"kubernetes.io/projected/fbbb9af9-1713-44f4-afcd-901667677352-kube-api-access-xpq6x\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.156151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.156032    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbb9af9-1713-44f4-afcd-901667677352-tls-certs\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.156151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.156106    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-tmp-dir\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.156151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.156128    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-model-cache\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.156388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.156152    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-dshm\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.156443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.156423    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-home\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.156506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.156490    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-tmp-dir\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.156563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.156531    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-model-cache\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.158459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.158435    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-dshm\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.158637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.158620    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbb9af9-1713-44f4-afcd-901667677352-tls-certs\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.165449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.165428    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpq6x\" (UniqueName: \"kubernetes.io/projected/fbbb9af9-1713-44f4-afcd-901667677352-kube-api-access-xpq6x\") pod \"simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") " pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.258274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.258195    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:47.321245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.320289    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:47.553443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:47.553414    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"]
Nov 03 13:18:47.555118 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:47.555083    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbbb9af9_1713_44f4_afcd_901667677352.slice/crio-1f6cf6f58940f9ee745cc7a1220bd8ba5f5bac9645513aacacd70f45574a1d3d WatchSource:0}: Error finding container 1f6cf6f58940f9ee745cc7a1220bd8ba5f5bac9645513aacacd70f45574a1d3d: Status 404 returned error can't find the container with id 1f6cf6f58940f9ee745cc7a1220bd8ba5f5bac9645513aacacd70f45574a1d3d
Nov 03 13:18:48.031343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:48.031296    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:48.304394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:48.304305    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" event={"ID":"fbbb9af9-1713-44f4-afcd-901667677352","Type":"ContainerStarted","Data":"8b93dc9d868d44062d10e5d2bb91229f923a55c36595551e5c516b1ea64e16ec"}
Nov 03 13:18:48.304394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:48.304350    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" event={"ID":"fbbb9af9-1713-44f4-afcd-901667677352","Type":"ContainerStarted","Data":"1f6cf6f58940f9ee745cc7a1220bd8ba5f5bac9645513aacacd70f45574a1d3d"}
Nov 03 13:18:48.325378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:48.325317    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podStartSLOduration=2.3253032940000002 podStartE2EDuration="2.325303294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:48.322865112 +0000 UTC m=+569.400630138" watchObservedRunningTime="2026-11-03 13:18:48.325303294 +0000 UTC m=+569.403068319"
Nov 03 13:18:48.724138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:48.724100    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:49.258341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:49.258306    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:49.512973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:49.512872    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:49.567855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:49.567824    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:49.580303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:49.580279    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:50.158586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:50.158550    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:50.331415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:50.331389    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:18:50.969720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:50.969637    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:51.609737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:51.609702    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:52.214559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.214524    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:52.402613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.402576    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:52.415186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.415161    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:52.640785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.640693    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"]
Nov 03 13:18:52.647435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.647413    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.654704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.654679    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"premium-w2-e440bc-kserve-self-signed-certs\""
Nov 03 13:18:52.657702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.657670    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"]
Nov 03 13:18:52.717142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.717107    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-tmp-dir\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.717142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.717144    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8d910c-8088-4ad9-95ca-571f9af28bea-tls-certs\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.717414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.717234    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-home\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.717414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.717313    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2fq\" (UniqueName: \"kubernetes.io/projected/9a8d910c-8088-4ad9-95ca-571f9af28bea-kube-api-access-9d2fq\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.717414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.717360    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-dshm\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.717521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.717466    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-model-cache\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.719789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.719763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:52.826471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.825403    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-tmp-dir\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.826471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.825461    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8d910c-8088-4ad9-95ca-571f9af28bea-tls-certs\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.826471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.825515    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-home\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.826471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.825545    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2fq\" (UniqueName: \"kubernetes.io/projected/9a8d910c-8088-4ad9-95ca-571f9af28bea-kube-api-access-9d2fq\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.826471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.825579    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-dshm\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.826471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.825661    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-model-cache\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.826927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.826497    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-home\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.826927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.826739    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-tmp-dir\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.838270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.827534    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-model-cache\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.838270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.834308    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-dshm\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.838270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.834631    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8d910c-8088-4ad9-95ca-571f9af28bea-tls-certs\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.843028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.842996    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2fq\" (UniqueName: \"kubernetes.io/projected/9a8d910c-8088-4ad9-95ca-571f9af28bea-kube-api-access-9d2fq\") pod \"premium-w2-e440bc-kserve-67699b949d-6bwsg\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") " pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:52.855815 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.855781    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:52.961648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:52.961615    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:53.239102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:53.239077    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"]
Nov 03 13:18:53.241350 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:53.241319    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a8d910c_8088_4ad9_95ca_571f9af28bea.slice/crio-8d2d9b1832c7df8ddcb2ed96e69d366b9fc9f60043c8710f10cc88c6b5f978a4 WatchSource:0}: Error finding container 8d2d9b1832c7df8ddcb2ed96e69d366b9fc9f60043c8710f10cc88c6b5f978a4: Status 404 returned error can't find the container with id 8d2d9b1832c7df8ddcb2ed96e69d366b9fc9f60043c8710f10cc88c6b5f978a4
Nov 03 13:18:53.339837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:53.339799    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" event={"ID":"9a8d910c-8088-4ad9-95ca-571f9af28bea","Type":"ContainerStarted","Data":"097732f26d4575736cc1c57c48bec4b2e5f53f97c5e08dc328ed857b7b4f244c"}
Nov 03 13:18:53.340036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:53.339851    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" event={"ID":"9a8d910c-8088-4ad9-95ca-571f9af28bea","Type":"ContainerStarted","Data":"8d2d9b1832c7df8ddcb2ed96e69d366b9fc9f60043c8710f10cc88c6b5f978a4"}
Nov 03 13:18:53.359709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:53.359680    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:18:53.366177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:53.366131    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podStartSLOduration=1.366117122 podStartE2EDuration="1.366117122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:53.364589728 +0000 UTC m=+574.442354755" watchObservedRunningTime="2026-11-03 13:18:53.366117122 +0000 UTC m=+574.443882149"
Nov 03 13:18:53.512393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:53.512298    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:53.962190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:53.962156    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:18:54.116968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:54.116931    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:54.459757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:54.459726    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:18:54.731304 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:18:54.731223    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:18:54.859449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:54.859414    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:55.587661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:55.587634    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:56.312072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:56.312038    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:57.092050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:57.092018    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:57.258638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:57.258598    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:57.271718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:57.271687    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:57.376825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:57.376741    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:18:57.809253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:57.809196    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:58.043133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.043100    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"]
Nov 03 13:18:58.047053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.047032    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.056238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.056198    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c381a7\"/\"premium-w4-c381a7-kserve-self-signed-certs\""
Nov 03 13:18:58.057400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.057374    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"]
Nov 03 13:18:58.127522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.127429    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:58.180963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.180813    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6cgs\" (UniqueName: \"kubernetes.io/projected/68c430da-26cd-420a-85de-84e4187fe5d0-kube-api-access-t6cgs\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.181230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.181192    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c430da-26cd-420a-85de-84e4187fe5d0-tls-certs\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.181453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.181436    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-dshm\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.181597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.181582    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-model-cache\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.181783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.181766    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-tmp-dir\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.181905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.181890    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-home\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.282822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.282786    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-dshm\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.283029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.282846    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-model-cache\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.283029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.282899    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-tmp-dir\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.283029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.282923    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-home\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.283029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.282955    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cgs\" (UniqueName: \"kubernetes.io/projected/68c430da-26cd-420a-85de-84e4187fe5d0-kube-api-access-t6cgs\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.283029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.282985    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c430da-26cd-420a-85de-84e4187fe5d0-tls-certs\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.283412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.283386    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-model-cache\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.283482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.283425    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-tmp-dir\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.283528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.283494    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-home\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.285189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.285163    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-dshm\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.285462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.285444    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c430da-26cd-420a-85de-84e4187fe5d0-tls-certs\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.291227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.291197    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cgs\" (UniqueName: \"kubernetes.io/projected/68c430da-26cd-420a-85de-84e4187fe5d0-kube-api-access-t6cgs\") pod \"premium-w4-c381a7-kserve-66b85f8b97-cdhpp\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") " pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.361826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.361797    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:18:58.510724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.510675    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:58.640759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:58.640735    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"]
Nov 03 13:18:58.642690 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:18:58.642662    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68c430da_26cd_420a_85de_84e4187fe5d0.slice/crio-cbf721fddcd776b2edebcb8a693302ab44124580fde37eee21f8475e8419992b WatchSource:0}: Error finding container cbf721fddcd776b2edebcb8a693302ab44124580fde37eee21f8475e8419992b: Status 404 returned error can't find the container with id cbf721fddcd776b2edebcb8a693302ab44124580fde37eee21f8475e8419992b
Nov 03 13:18:59.186854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:59.186806    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:59.335873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:59.335833    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:59.380195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:59.380159    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" event={"ID":"68c430da-26cd-420a-85de-84e4187fe5d0","Type":"ContainerStarted","Data":"252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da"}
Nov 03 13:18:59.380195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:59.380199    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" event={"ID":"68c430da-26cd-420a-85de-84e4187fe5d0","Type":"ContainerStarted","Data":"cbf721fddcd776b2edebcb8a693302ab44124580fde37eee21f8475e8419992b"}
Nov 03 13:18:59.401012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:59.400956    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podStartSLOduration=1.400942469 podStartE2EDuration="1.400942469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:59.399350354 +0000 UTC m=+580.477115381" watchObservedRunningTime="2026-11-03 13:18:59.400942469 +0000 UTC m=+580.478707494"
Nov 03 13:18:59.763553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:59.763498    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:59.899600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:18:59.899571    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:00.083473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:00.083383    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:00.362986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:00.362906    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:19:00.788352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:00.788320    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:01.495917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:01.495887    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:02.113048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:02.112985    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:02.962071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:02.962041    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:19:02.975162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:02.975136    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:19:03.218131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:03.218049    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:03.415247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:03.415198    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:19:03.601898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:03.601811    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:03.736895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:03.736866    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:04.304790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.304756    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:04.439136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.439098    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"]
Nov 03 13:19:04.443118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.443098    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.455581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.455553    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"]
Nov 03 13:19:04.482629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.482594    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a4ab8e\"/\"premium-w0-a4ab8e-kserve-self-signed-certs\""
Nov 03 13:19:04.517635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.517609    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:04.548161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.548124    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-home\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.548383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.548169    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/495bb85e-709f-405d-a92e-e681113bb0e1-kube-api-access-8dqg5\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.548383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.548297    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-model-cache\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.548383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.548355    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-dshm\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.548502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.548461    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-tmp-dir\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.548629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.548571    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495bb85e-709f-405d-a92e-e681113bb0e1-tls-certs\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.650491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.650393    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-home\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.650491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.650441    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/495bb85e-709f-405d-a92e-e681113bb0e1-kube-api-access-8dqg5\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.650491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.650487    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-model-cache\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.650778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.650521    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-dshm\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.650778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.650569    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-tmp-dir\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.650778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.650651    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495bb85e-709f-405d-a92e-e681113bb0e1-tls-certs\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.650898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.650874    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-home\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.650944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.650928    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-model-cache\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.651005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.650981    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-tmp-dir\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.653407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.653386    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495bb85e-709f-405d-a92e-e681113bb0e1-tls-certs\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.653539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.653524    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-dshm\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.659854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.659818    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/495bb85e-709f-405d-a92e-e681113bb0e1-kube-api-access-8dqg5\") pod \"premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") " pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.758047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.758014    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:04.976440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.976406    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"]
Nov 03 13:19:04.981359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.981336    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:04.993475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.993449    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-17adbf-dockercfg-wjxr7\""
Nov 03 13:19:04.994575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:04.994553    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"]
Nov 03 13:19:05.016098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.016062    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:05.049911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.049883    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"]
Nov 03 13:19:05.051812 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:19:05.051781    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod495bb85e_709f_405d_a92e_e681113bb0e1.slice/crio-d1ca8a3229a4ac9a8ea89060fa42c011d931fa7675fc002e8817d6b3ef5a6795 WatchSource:0}: Error finding container d1ca8a3229a4ac9a8ea89060fa42c011d931fa7675fc002e8817d6b3ef5a6795: Status 404 returned error can't find the container with id d1ca8a3229a4ac9a8ea89060fa42c011d931fa7675fc002e8817d6b3ef5a6795
Nov 03 13:19:05.058061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.058037    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-17adbf\""
Nov 03 13:19:05.160235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.159302    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnn7c\" (UniqueName: \"kubernetes.io/projected/57ef622b-2fc2-4167-a663-d61a96eddca9-kube-api-access-wnn7c\") pod \"payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg\" (UID: \"57ef622b-2fc2-4167-a663-d61a96eddca9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:05.160235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.159383    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ef622b-2fc2-4167-a663-d61a96eddca9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg\" (UID: \"57ef622b-2fc2-4167-a663-d61a96eddca9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:05.160514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.160331    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:05.260631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.260540    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ef622b-2fc2-4167-a663-d61a96eddca9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg\" (UID: \"57ef622b-2fc2-4167-a663-d61a96eddca9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:05.260799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.260739    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnn7c\" (UniqueName: \"kubernetes.io/projected/57ef622b-2fc2-4167-a663-d61a96eddca9-kube-api-access-wnn7c\") pod \"payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg\" (UID: \"57ef622b-2fc2-4167-a663-d61a96eddca9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:05.261426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.261408    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ef622b-2fc2-4167-a663-d61a96eddca9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg\" (UID: \"57ef622b-2fc2-4167-a663-d61a96eddca9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:05.269043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.269025    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnn7c\" (UniqueName: \"kubernetes.io/projected/57ef622b-2fc2-4167-a663-d61a96eddca9-kube-api-access-wnn7c\") pod \"payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg\" (UID: \"57ef622b-2fc2-4167-a663-d61a96eddca9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:05.296022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.295974    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:05.419088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.419052    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" event={"ID":"495bb85e-709f-405d-a92e-e681113bb0e1","Type":"ContainerStarted","Data":"8c697e6621871ca1721c3a4d494a5f7fc6bcec34fdc5b5e86ef53c9685a586d6"}
Nov 03 13:19:05.419592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.419099    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" event={"ID":"495bb85e-709f-405d-a92e-e681113bb0e1","Type":"ContainerStarted","Data":"d1ca8a3229a4ac9a8ea89060fa42c011d931fa7675fc002e8817d6b3ef5a6795"}
Nov 03 13:19:05.437256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.437196    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podStartSLOduration=1.437183019 podStartE2EDuration="1.437183019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:05.435420164 +0000 UTC m=+586.513185190" watchObservedRunningTime="2026-11-03 13:19:05.437183019 +0000 UTC m=+586.514948044"
Nov 03 13:19:05.579925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.579901    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"]
Nov 03 13:19:05.582522 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:19:05.582490    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ef622b_2fc2_4167_a663_d61a96eddca9.slice/crio-9b9776cff959c7f32aa40f714d1b54f78e15f936f9e84b3414261a0198d4ab54 WatchSource:0}: Error finding container 9b9776cff959c7f32aa40f714d1b54f78e15f936f9e84b3414261a0198d4ab54: Status 404 returned error can't find the container with id 9b9776cff959c7f32aa40f714d1b54f78e15f936f9e84b3414261a0198d4ab54
Nov 03 13:19:05.759125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.759094    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:05.814418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.814378    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:05.947464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:05.947385    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:06.426290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:06.426250    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg" event={"ID":"57ef622b-2fc2-4167-a663-d61a96eddca9","Type":"ContainerStarted","Data":"63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771"}
Nov 03 13:19:06.426745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:06.426314    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg" event={"ID":"57ef622b-2fc2-4167-a663-d61a96eddca9","Type":"ContainerStarted","Data":"9b9776cff959c7f32aa40f714d1b54f78e15f936f9e84b3414261a0198d4ab54"}
Nov 03 13:19:06.426745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:06.426502    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:06.445754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:06.445694    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg" podStartSLOduration=2.445681218 podStartE2EDuration="2.445681218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:06.442937612 +0000 UTC m=+587.520702637" watchObservedRunningTime="2026-11-03 13:19:06.445681218 +0000 UTC m=+587.523446244"
Nov 03 13:19:06.476948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:06.476902    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:06.937977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:06.937943    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:19:07.161903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:07.161867    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:07.819853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:07.819817    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:08.362173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:08.362144    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:19:08.375262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:08.375235    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:19:08.455163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:08.455123    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:19:08.464804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:08.464774    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:09.107034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:09.107000    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:10.834570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:10.834537    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:10.965403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:10.965343    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:11.104907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:11.104800    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:11.715561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:11.715527    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:12.317029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:12.316996    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:12.903002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:12.902971    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:13.546447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:13.546412    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:14.496752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:14.496721    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:14.758596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:14.758518    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:14.771342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:14.771311    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:15.391117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:15.391084    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:15.492405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:15.492378    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:19:16.090459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:16.090419    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:17.125932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:17.125903    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:17.262391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:17.262356    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:17.441523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:17.441434    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:19:17.617062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:17.616984    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:18.187709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:18.187672    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:18.462598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:18.462567    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:19:18.484427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:18.484391    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:18.841641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:18.841556    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:18.975736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:18.975706    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:19.157893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.157817    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:19:19.160394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.160372    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:19:19.283151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.283114    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"]
Nov 03 13:19:19.291227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.291193    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.301320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.301292    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"]
Nov 03 13:19:19.312295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.312268    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-podinfo\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.312432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.312309    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-certs\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.312494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.312426    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-socket\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.312545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.312512    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-data\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.312609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.312588    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb690a-f1c0-4871-8c1b-203584b7645e-istiod-ca-cert\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.312665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.312644    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-envoy\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.312742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.312723    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-credential-socket\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.312794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.312779    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-token\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.312853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.312815    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62whp\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-kube-api-access-62whp\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.327456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.327431    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-db71f4-openshift-default-dockercfg-vg5bc\""
Nov 03 13:19:19.390706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.390672    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:19.413726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.413646    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62whp\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-kube-api-access-62whp\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.413726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.413700    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-podinfo\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.413948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.413870    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-certs\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.413951    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-socket\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.413987    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-data\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.414017    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb690a-f1c0-4871-8c1b-203584b7645e-istiod-ca-cert\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.414060    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-envoy\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.414158    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-credential-socket\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.414184    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-token\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.414326    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-certs\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.414740    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-data\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.414748    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb690a-f1c0-4871-8c1b-203584b7645e-istiod-ca-cert\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.414787    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-socket\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.414896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.414849    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-credential-socket\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.416483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.416458    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-podinfo\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.416873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.416855    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-envoy\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.428731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.428700    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-token\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.428990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.428973    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62whp\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-kube-api-access-62whp\") pod \"e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") " pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.542050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.542007    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:19.608187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.608150    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:19.835939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.835906    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:19.891257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.891235    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"]
Nov 03 13:19:19.892899 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:19:19.892869    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cb690a_f1c0_4871_8c1b_203584b7645e.slice/crio-6e039d83c9386448be1599e91e8ef8ad745a9849263e718b7136d5cb9bbedb0f WatchSource:0}: Error finding container 6e039d83c9386448be1599e91e8ef8ad745a9849263e718b7136d5cb9bbedb0f: Status 404 returned error can't find the container with id 6e039d83c9386448be1599e91e8ef8ad745a9849263e718b7136d5cb9bbedb0f
Nov 03 13:19:19.895361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.895327    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:19.895444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.895392    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:19.895444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:19.895421    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:20.516120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:20.516086    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" event={"ID":"d2cb690a-f1c0-4871-8c1b-203584b7645e","Type":"ContainerStarted","Data":"711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718"}
Nov 03 13:19:20.516555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:20.516144    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" event={"ID":"d2cb690a-f1c0-4871-8c1b-203584b7645e","Type":"ContainerStarted","Data":"6e039d83c9386448be1599e91e8ef8ad745a9849263e718b7136d5cb9bbedb0f"}
Nov 03 13:19:20.538861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:20.538807    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" podStartSLOduration=1.538791563 podStartE2EDuration="1.538791563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:20.536904377 +0000 UTC m=+601.614669403" watchObservedRunningTime="2026-11-03 13:19:20.538791563 +0000 UTC m=+601.616556590"
Nov 03 13:19:20.608519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:20.608486    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:20.612731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:20.612699    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:20.613542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:20.613525    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:21.310533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:21.310499    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:21.522346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:21.522320    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:21.523352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:21.523334    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:22.180099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:22.180055    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:22.899804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:22.899770    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:23.064877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:23.064830    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:23.699966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:23.699909    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:24.162995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.162944    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:24.258811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.258777    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"]
Nov 03 13:19:24.262692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.262675    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:24.276293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.276267    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"]
Nov 03 13:19:24.298992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.298955    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:24.304002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.303977    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-db71f4\""
Nov 03 13:19:24.304581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.304544    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-db71f4-dockercfg-2m5zc\""
Nov 03 13:19:24.364903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.364867    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mrv\" (UniqueName: \"kubernetes.io/projected/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-kube-api-access-v6mrv\") pod \"payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj\" (UID: \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:24.365120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.365006    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj\" (UID: \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:24.466480 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.466454    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj\" (UID: \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:24.466654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.466556    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mrv\" (UniqueName: \"kubernetes.io/projected/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-kube-api-access-v6mrv\") pod \"payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj\" (UID: \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:24.467167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.467148    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj\" (UID: \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:24.475542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.475512    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mrv\" (UniqueName: \"kubernetes.io/projected/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-kube-api-access-v6mrv\") pod \"payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj\" (UID: \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:24.576681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.576652    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:24.705524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.705490    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:24.737029 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:19:24.736946    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:19:24.854821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:24.854796    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"]
Nov 03 13:19:24.856951 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:19:24.856924    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b89cf0_3d7f_486b_bd21_fe8193a2ea89.slice/crio-fea2d92ed7a12dcd7e1e1151e1b7d06dcf0c884b6481cf1b21261cfd974ee2ec WatchSource:0}: Error finding container fea2d92ed7a12dcd7e1e1151e1b7d06dcf0c884b6481cf1b21261cfd974ee2ec: Status 404 returned error can't find the container with id fea2d92ed7a12dcd7e1e1151e1b7d06dcf0c884b6481cf1b21261cfd974ee2ec
Nov 03 13:19:25.419689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:25.419652    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:25.547068 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:25.547036    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj" event={"ID":"41b89cf0-3d7f-486b-bd21-fe8193a2ea89","Type":"ContainerStarted","Data":"e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e"}
Nov 03 13:19:25.547244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:25.547084    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj" event={"ID":"41b89cf0-3d7f-486b-bd21-fe8193a2ea89","Type":"ContainerStarted","Data":"fea2d92ed7a12dcd7e1e1151e1b7d06dcf0c884b6481cf1b21261cfd974ee2ec"}
Nov 03 13:19:25.547244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:25.547105    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:25.563057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:25.563006    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj" podStartSLOduration=1.562992921 podStartE2EDuration="1.562992921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:25.562641746 +0000 UTC m=+606.640406779" watchObservedRunningTime="2026-11-03 13:19:25.562992921 +0000 UTC m=+606.640757946"
Nov 03 13:19:26.010788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:26.010755    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:26.150850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:26.150805    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:26.555109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:26.555079    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:26.825995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:26.825897    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:27.822056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:27.822024    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:28.511310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:28.511084    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:29.208456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:29.208400    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:29.819970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:29.819937    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:30.430000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:30.429970    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:31.031496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.031438    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:31.295807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.295730    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"]
Nov 03 13:19:31.299760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.299739    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.307226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.307192    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c381a7-openshift-default-dockercfg-c5nmd\""
Nov 03 13:19:31.311986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.311961    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"]
Nov 03 13:19:31.343130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.343090    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.343329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.343190    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.343384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.343334    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.343384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.343362    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6pnc\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-kube-api-access-x6pnc\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.343488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.343390    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.343544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.343524    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.343607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.343591    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.343659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.343628    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.343730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.343713    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.384856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.384819    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"]
Nov 03 13:19:31.388700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.388683    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.406994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.406966    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a4ab8e-openshift-default-dockercfg-dm758\""
Nov 03 13:19:31.407914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.407892    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"]
Nov 03 13:19:31.444473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.444440    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.444924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.444572    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdc84\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-kube-api-access-hdc84\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.444924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.444612    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.444924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.444663    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.444924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.444729    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.444924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.444763    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.444924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.444806    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.445273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.444927    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.444976    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6pnc\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-kube-api-access-x6pnc\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445009    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.445273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445047    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445085    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.445273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445149    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445188    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445278    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445326    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445337    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.445569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445417    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445444    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.445569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445475    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445512    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445757    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.445908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.445763    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.447644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.447602    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.447743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.447709    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.454510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.454486    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.454945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.454925    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6pnc\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-kube-api-access-x6pnc\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.546663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.546587    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.546806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.546666    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.546806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.546690    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.546806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.546721    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.546806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.546739    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.546806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.546773    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.547070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.546842    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc84\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-kube-api-access-hdc84\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.547070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.546868    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.547070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.546886    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.547246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.547154    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.547246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.547185    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.547421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.547400    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.547499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.547407    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.547866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.547839    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.549404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.549381    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.549658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.549640    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.555612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.555588    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.555715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.555649    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdc84\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-kube-api-access-hdc84\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.616093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.616063    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:31.712056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.710463    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:31.720121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.720087    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:31.911501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.911475    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"]
Nov 03 13:19:31.913957 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:19:31.913928    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a0ce67_5f7d_4bce_afe1_ab40efbf422c.slice/crio-fdc2a0ba9046450db78cb12aa7bd4eefb5850a975a1eeea7eec7f7096e308a42 WatchSource:0}: Error finding container fdc2a0ba9046450db78cb12aa7bd4eefb5850a975a1eeea7eec7f7096e308a42: Status 404 returned error can't find the container with id fdc2a0ba9046450db78cb12aa7bd4eefb5850a975a1eeea7eec7f7096e308a42
Nov 03 13:19:31.916548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.916514    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:31.916627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.916584    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:31.916627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.916612    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:31.957744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:31.957722    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:19:32.011488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.011459    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"]
Nov 03 13:19:32.014484 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:19:32.014453    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c5bca5_53b9_475e_a4d3_c7fa153c4c35.slice/crio-eac143864edf9ce696afaeedd7eceae9aedabcdfd8f759d1d9b105aed145d547 WatchSource:0}: Error finding container eac143864edf9ce696afaeedd7eceae9aedabcdfd8f759d1d9b105aed145d547: Status 404 returned error can't find the container with id eac143864edf9ce696afaeedd7eceae9aedabcdfd8f759d1d9b105aed145d547
Nov 03 13:19:32.036498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.036458    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:32.036661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.036545    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:32.036661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.036591    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:32.519256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.518644    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:32.592077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.592040    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" event={"ID":"25c5bca5-53b9-475e-a4d3-c7fa153c4c35","Type":"ContainerStarted","Data":"8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450"}
Nov 03 13:19:32.592323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.592122    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" event={"ID":"25c5bca5-53b9-475e-a4d3-c7fa153c4c35","Type":"ContainerStarted","Data":"eac143864edf9ce696afaeedd7eceae9aedabcdfd8f759d1d9b105aed145d547"}
Nov 03 13:19:32.593916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.593888    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" event={"ID":"02a0ce67-5f7d-4bce-afe1-ab40efbf422c","Type":"ContainerStarted","Data":"82f58f5e7a3b97a73fe6199a2380d42d7b5a7c8d8764fc45a57728ad8098f761"}
Nov 03 13:19:32.594048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.593931    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" event={"ID":"02a0ce67-5f7d-4bce-afe1-ab40efbf422c","Type":"ContainerStarted","Data":"fdc2a0ba9046450db78cb12aa7bd4eefb5850a975a1eeea7eec7f7096e308a42"}
Nov 03 13:19:32.614189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.614139    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" podStartSLOduration=1.614127289 podStartE2EDuration="1.614127289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:32.613804302 +0000 UTC m=+613.691569329" watchObservedRunningTime="2026-11-03 13:19:32.614127289 +0000 UTC m=+613.691892309"
Nov 03 13:19:32.616399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.616381    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:32.618094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.618066    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:19:32.618148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.618116    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 13:19:32.670234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.654373    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" podStartSLOduration=1.654354101 podStartE2EDuration="1.654354101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:32.64580342 +0000 UTC m=+613.723568624" watchObservedRunningTime="2026-11-03 13:19:32.654354101 +0000 UTC m=+613.732119129"
Nov 03 13:19:32.710972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:32.710939    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:33.205956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:33.203022    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:33.616938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:33.616855    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:19:33.616938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:33.616908    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 13:19:33.712930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:33.712883    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:19:33.713104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:33.712942    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 13:19:33.939291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:33.933701    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:34.102244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:34.102117    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:34.597238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:34.591851    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:34.616569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:34.616535    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:19:34.616745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:34.616586    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 13:19:34.711359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:34.711320    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:19:34.711789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:34.711371    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 13:19:35.288235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:35.287422    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:35.617338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:35.617247    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:19:35.617338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:35.617299    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 13:19:35.711436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:35.711408    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:19:35.711842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:35.711456    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 13:19:36.085401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.085367    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:36.602113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.602081    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:19:36.616546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.616514    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:19:36.616715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.616564    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 13:19:36.711122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.711084    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:19:36.711360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.711149    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 13:19:36.817611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.817577    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"]
Nov 03 13:19:36.841492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.841433    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:36.901307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.901202    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"]
Nov 03 13:19:36.901620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.901600    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:36.910232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.910120    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:19:36.910359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.910253    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:19:36.910422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.910407    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:19:36.910463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:36.910444    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:19:37.018966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.018934    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b26e8d6-49bmt\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.019239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.019196    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b26e8d6-49bmt\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.019427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.019415    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2lc\" (UniqueName: \"kubernetes.io/projected/977ab5f9-3315-417e-9c00-02e4c38a012e-kube-api-access-df2lc\") pod \"maas-api-revoke-sub-0b26e8d6-49bmt\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.120191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.120149    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b26e8d6-49bmt\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.120414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.120239    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b26e8d6-49bmt\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.120414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.120367    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2lc\" (UniqueName: \"kubernetes.io/projected/977ab5f9-3315-417e-9c00-02e4c38a012e-kube-api-access-df2lc\") pod \"maas-api-revoke-sub-0b26e8d6-49bmt\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.124233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.121100    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b26e8d6-49bmt\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.144412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.144377    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2lc\" (UniqueName: \"kubernetes.io/projected/977ab5f9-3315-417e-9c00-02e4c38a012e-kube-api-access-df2lc\") pod \"maas-api-revoke-sub-0b26e8d6-49bmt\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.163283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.161049    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b26e8d6-49bmt\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.225672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.224409    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:37.405264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.402576    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:37.617981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.617942    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:19:37.618164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.618001    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 13:19:37.713176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.712714    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:19:37.713176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.712768    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 13:19:37.765274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.761779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:37.901290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:37.892702    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"]
Nov 03 13:19:37.903726 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:19:37.903700    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977ab5f9_3315_417e_9c00_02e4c38a012e.slice/crio-829772d04ef99362b0693632fd12c7f0d820f1a1182e35ca9192b6c4b0692b56 WatchSource:0}: Error finding container 829772d04ef99362b0693632fd12c7f0d820f1a1182e35ca9192b6c4b0692b56: Status 404 returned error can't find the container with id 829772d04ef99362b0693632fd12c7f0d820f1a1182e35ca9192b6c4b0692b56
Nov 03 13:19:38.048289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:38.044081    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:38.617805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:38.617304    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 13:19:38.617805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:38.617371    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 13:19:38.711421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:38.711388    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 13:19:38.711704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:38.711667    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 13:19:38.725724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:38.725687    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt" event={"ID":"977ab5f9-3315-417e-9c00-02e4c38a012e","Type":"ContainerStarted","Data":"18fa6b782137c494936efad535baf923e1d2c5a1de8669b1a9093dc6b6b7f556"}
Nov 03 13:19:38.725876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:38.725755    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt" event={"ID":"977ab5f9-3315-417e-9c00-02e4c38a012e","Type":"ContainerStarted","Data":"829772d04ef99362b0693632fd12c7f0d820f1a1182e35ca9192b6c4b0692b56"}
Nov 03 13:19:38.751604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:38.751541    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt" podStartSLOduration=2.7515197479999998 podStartE2EDuration="2.751519748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:38.746162365 +0000 UTC m=+619.823927393" watchObservedRunningTime="2026-11-03 13:19:38.751519748 +0000 UTC m=+619.829284775"
Nov 03 13:19:38.755718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:38.755645    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:39.080999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.080963    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:39.622831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.622800    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:39.623119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.623101    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:39.623782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.623758    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:19:39.716662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.716626    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:39.717063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.717037    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:39.717931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.717911    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:19:39.735855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.735814    5420 generic.go:356] "Generic (PLEG): container finished" podID="977ab5f9-3315-417e-9c00-02e4c38a012e" containerID="18fa6b782137c494936efad535baf923e1d2c5a1de8669b1a9093dc6b6b7f556" exitCode=0
Nov 03 13:19:39.736047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.735918    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt" event={"ID":"977ab5f9-3315-417e-9c00-02e4c38a012e","Type":"ContainerDied","Data":"18fa6b782137c494936efad535baf923e1d2c5a1de8669b1a9093dc6b6b7f556"}
Nov 03 13:19:39.927400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:39.924504    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:40.730285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:40.730235    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:40.890689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:40.890665    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:40.912615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:40.912577    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-xks-ca\") pod \"977ab5f9-3315-417e-9c00-02e4c38a012e\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") "
Nov 03 13:19:40.912762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:40.912737    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-openshift-ca\") pod \"977ab5f9-3315-417e-9c00-02e4c38a012e\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") "
Nov 03 13:19:40.912849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:40.912781    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977ab5f9-3315-417e-9c00-02e4c38a012e-kube-api-access-df2lc\" (UniqueName: \"kubernetes.io/projected/977ab5f9-3315-417e-9c00-02e4c38a012e-kube-api-access-df2lc\") pod \"977ab5f9-3315-417e-9c00-02e4c38a012e\" (UID: \"977ab5f9-3315-417e-9c00-02e4c38a012e\") "
Nov 03 13:19:40.913362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:40.913316    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-openshift-ca" pod "977ab5f9-3315-417e-9c00-02e4c38a012e" (UID: "977ab5f9-3315-417e-9c00-02e4c38a012e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:40.915517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:40.915494    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-xks-ca" pod "977ab5f9-3315-417e-9c00-02e4c38a012e" (UID: "977ab5f9-3315-417e-9c00-02e4c38a012e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:40.915691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:40.915668    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977ab5f9-3315-417e-9c00-02e4c38a012e-kube-api-access-df2lc" pod "977ab5f9-3315-417e-9c00-02e4c38a012e" (UID: "977ab5f9-3315-417e-9c00-02e4c38a012e"). InnerVolumeSpecName "kube-api-access-df2lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:41.014403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:41.014325    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:41.014403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:41.014352    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977ab5f9-3315-417e-9c00-02e4c38a012e-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:41.014403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:41.014363    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df2lc\" (UniqueName: \"kubernetes.io/projected/977ab5f9-3315-417e-9c00-02e4c38a012e-kube-api-access-df2lc\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:41.283161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:41.283074    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:19:41.397472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:41.397437    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:41.750873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:41.750843    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"
Nov 03 13:19:41.751306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:41.750868    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt" event={"ID":"977ab5f9-3315-417e-9c00-02e4c38a012e","Type":"ContainerDied","Data":"829772d04ef99362b0693632fd12c7f0d820f1a1182e35ca9192b6c4b0692b56"}
Nov 03 13:19:41.751306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:41.750966    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="829772d04ef99362b0693632fd12c7f0d820f1a1182e35ca9192b6c4b0692b56"
Nov 03 13:19:42.165605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:42.165522    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:42.760414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:42.760373    5420 generic.go:356] "Generic (PLEG): container finished" podID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerID="711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718" exitCode=0
Nov 03 13:19:42.760933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:42.760445    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" event={"ID":"d2cb690a-f1c0-4871-8c1b-203584b7645e","Type":"ContainerDied","Data":"711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718"}
Nov 03 13:19:42.760933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:42.760869    5420 scope.go:122] "RemoveContainer" containerID="711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718"
Nov 03 13:19:42.762276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:42.762248    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:42.762402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:42.762322    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:42.762402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:42.762366    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:19:42.804433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:42.804394    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:43.128431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:43.128355    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:43.703692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:43.703658    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:43.767864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:43.767828    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" event={"ID":"d2cb690a-f1c0-4871-8c1b-203584b7645e","Type":"ContainerStarted","Data":"a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6"}
Nov 03 13:19:43.975131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:43.975046    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:44.429489 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:44.425748    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:44.609273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:44.609239    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:44.609456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:44.609396    5420 patch_prober.go:29] interesting pod/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 13:19:44.609456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:44.609440    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 13:19:45.115816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:45.115788    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:45.608728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:45.608694    5420 patch_prober.go:29] interesting pod/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 13:19:45.608914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:45.608757    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 13:19:45.705571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:45.705530    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:46.518896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:46.518855    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:46.609414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:46.609379    5420 patch_prober.go:29] interesting pod/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 13:19:46.609588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:46.609436    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 13:19:47.201109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.201059    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:47.532975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.532934    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"]
Nov 03 13:19:47.542526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.542493    5420 cpu_manager.go:414] "containerMap: removing container" podUID="977ab5f9-3315-417e-9c00-02e4c38a012e" containerName="revoke-keys"
Nov 03 13:19:47.542706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.542689    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977ab5f9-3315-417e-9c00-02e4c38a012e" containerName="revoke-keys"
Nov 03 13:19:47.542940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.542924    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="977ab5f9-3315-417e-9c00-02e4c38a012e" containerName="revoke-keys"
Nov 03 13:19:47.550095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.550072    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.550711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.550677    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"]
Nov 03 13:19:47.554835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.554813    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:19:47.555875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.555728    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:19:47.555875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.555728    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:19:47.555875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.555842    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:19:47.609464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.609436    5420 patch_prober.go:29] interesting pod/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 13:19:47.609625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.609484    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 13:19:47.610992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.610965    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qksps\" (UniqueName: \"kubernetes.io/projected/23e481f0-0b30-4881-87ac-e0e96a798411-kube-api-access-qksps\") pod \"maas-api-revoke-sub-ca5a21fb-vnx2r\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.611119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.611018    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5a21fb-vnx2r\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.611175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.611155    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5a21fb-vnx2r\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.712460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.712423    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5a21fb-vnx2r\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.712665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.712520    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qksps\" (UniqueName: \"kubernetes.io/projected/23e481f0-0b30-4881-87ac-e0e96a798411-kube-api-access-qksps\") pod \"maas-api-revoke-sub-ca5a21fb-vnx2r\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.712665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.712560    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5a21fb-vnx2r\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.713164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.713144    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5a21fb-vnx2r\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.714834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.714816    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5a21fb-vnx2r\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.744237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.740920    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qksps\" (UniqueName: \"kubernetes.io/projected/23e481f0-0b30-4881-87ac-e0e96a798411-kube-api-access-qksps\") pod \"maas-api-revoke-sub-ca5a21fb-vnx2r\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.865722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.865636    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:47.896509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:47.892373    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:48.176637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:48.176608    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"]
Nov 03 13:19:48.177868 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:19:48.177840    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e481f0_0b30_4881_87ac_e0e96a798411.slice/crio-7140e724bc348d5015fa39a0f4c4b264b193f6e3f9740abb0d81322e22b66661 WatchSource:0}: Error finding container 7140e724bc348d5015fa39a0f4c4b264b193f6e3f9740abb0d81322e22b66661: Status 404 returned error can't find the container with id 7140e724bc348d5015fa39a0f4c4b264b193f6e3f9740abb0d81322e22b66661
Nov 03 13:19:48.519289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:48.519254    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:48.608799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:48.608761    5420 patch_prober.go:29] interesting pod/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 13:19:48.609187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:48.608814    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 13:19:48.798826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:48.798743    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r" event={"ID":"23e481f0-0b30-4881-87ac-e0e96a798411","Type":"ContainerStarted","Data":"d52e218e3f42ab743a80b05dd84a15381ca60f183b8828ebc4bb737be8b3082a"}
Nov 03 13:19:48.798826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:48.798791    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r" event={"ID":"23e481f0-0b30-4881-87ac-e0e96a798411","Type":"ContainerStarted","Data":"7140e724bc348d5015fa39a0f4c4b264b193f6e3f9740abb0d81322e22b66661"}
Nov 03 13:19:48.816366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:48.816312    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r" podStartSLOduration=1.8162981299999998 podStartE2EDuration="1.81629813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:48.81534594 +0000 UTC m=+629.893110967" watchObservedRunningTime="2026-11-03 13:19:48.81629813 +0000 UTC m=+629.894063156"
Nov 03 13:19:49.130825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.130796    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"]
Nov 03 13:19:49.134444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.134422    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.143265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.141753    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"]
Nov 03 13:19:49.203336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.203299    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:49.242123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.242088    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-169ee085-gfxqg\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.242329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.242152    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-169ee085-gfxqg\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.242329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.242290    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7qx7\" (UniqueName: \"kubernetes.io/projected/06cacf08-f842-4b04-b873-e55a01365aa5-kube-api-access-v7qx7\") pod \"maas-api-revoke-sub-169ee085-gfxqg\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.342954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.342922    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-169ee085-gfxqg\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.343287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.343264    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-169ee085-gfxqg\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.343491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.343474    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7qx7\" (UniqueName: \"kubernetes.io/projected/06cacf08-f842-4b04-b873-e55a01365aa5-kube-api-access-v7qx7\") pod \"maas-api-revoke-sub-169ee085-gfxqg\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.344132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.344091    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-169ee085-gfxqg\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.345983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.345961    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-169ee085-gfxqg\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.352760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.352741    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7qx7\" (UniqueName: \"kubernetes.io/projected/06cacf08-f842-4b04-b873-e55a01365aa5-kube-api-access-v7qx7\") pod \"maas-api-revoke-sub-169ee085-gfxqg\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.492925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.492894    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:49.613970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.613937    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:49.614483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.614368    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:49.615167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.615151    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:19:49.769151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.769125    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"]
Nov 03 13:19:49.770907 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:19:49.770877    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cacf08_f842_4b04_b873_e55a01365aa5.slice/crio-a49eddd7eb9d189e3295933df0f8c6c0bf86b53c36961bde19aaf14d3fa72e2b WatchSource:0}: Error finding container a49eddd7eb9d189e3295933df0f8c6c0bf86b53c36961bde19aaf14d3fa72e2b: Status 404 returned error can't find the container with id a49eddd7eb9d189e3295933df0f8c6c0bf86b53c36961bde19aaf14d3fa72e2b
Nov 03 13:19:49.806740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.806712    5420 generic.go:356] "Generic (PLEG): container finished" podID="23e481f0-0b30-4881-87ac-e0e96a798411" containerID="d52e218e3f42ab743a80b05dd84a15381ca60f183b8828ebc4bb737be8b3082a" exitCode=0
Nov 03 13:19:49.806889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.806801    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r" event={"ID":"23e481f0-0b30-4881-87ac-e0e96a798411","Type":"ContainerDied","Data":"d52e218e3f42ab743a80b05dd84a15381ca60f183b8828ebc4bb737be8b3082a"}
Nov 03 13:19:49.808162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.808136    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg" event={"ID":"06cacf08-f842-4b04-b873-e55a01365aa5","Type":"ContainerStarted","Data":"a49eddd7eb9d189e3295933df0f8c6c0bf86b53c36961bde19aaf14d3fa72e2b"}
Nov 03 13:19:49.906110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:49.906079    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:50.505180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:50.505105    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:50.815690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:50.815604    5420 generic.go:356] "Generic (PLEG): container finished" podID="06cacf08-f842-4b04-b873-e55a01365aa5" containerID="5f68363568338d31221b112fc247846b7bfdf93f995dfa6592a88a756a1aa94e" exitCode=0
Nov 03 13:19:50.816121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:50.815687    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg" event={"ID":"06cacf08-f842-4b04-b873-e55a01365aa5","Type":"ContainerDied","Data":"5f68363568338d31221b112fc247846b7bfdf93f995dfa6592a88a756a1aa94e"}
Nov 03 13:19:50.952614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:50.952590    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:51.064717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.064686    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-openshift-ca\") pod \"23e481f0-0b30-4881-87ac-e0e96a798411\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") "
Nov 03 13:19:51.064894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.064756    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e481f0-0b30-4881-87ac-e0e96a798411-kube-api-access-qksps\" (UniqueName: \"kubernetes.io/projected/23e481f0-0b30-4881-87ac-e0e96a798411-kube-api-access-qksps\") pod \"23e481f0-0b30-4881-87ac-e0e96a798411\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") "
Nov 03 13:19:51.064894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.064832    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-xks-ca\") pod \"23e481f0-0b30-4881-87ac-e0e96a798411\" (UID: \"23e481f0-0b30-4881-87ac-e0e96a798411\") "
Nov 03 13:19:51.065603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.065576    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-openshift-ca" pod "23e481f0-0b30-4881-87ac-e0e96a798411" (UID: "23e481f0-0b30-4881-87ac-e0e96a798411"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:51.086476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.081290    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-xks-ca" pod "23e481f0-0b30-4881-87ac-e0e96a798411" (UID: "23e481f0-0b30-4881-87ac-e0e96a798411"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:51.086476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.082480    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e481f0-0b30-4881-87ac-e0e96a798411-kube-api-access-qksps" pod "23e481f0-0b30-4881-87ac-e0e96a798411" (UID: "23e481f0-0b30-4881-87ac-e0e96a798411"). InnerVolumeSpecName "kube-api-access-qksps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:51.166981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.166936    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:51.166981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.166980    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e481f0-0b30-4881-87ac-e0e96a798411-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:51.167207 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.167001    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qksps\" (UniqueName: \"kubernetes.io/projected/23e481f0-0b30-4881-87ac-e0e96a798411-kube-api-access-qksps\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:51.417311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.417195    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:51.822657 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.822630    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"
Nov 03 13:19:51.823072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.822668    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r" event={"ID":"23e481f0-0b30-4881-87ac-e0e96a798411","Type":"ContainerDied","Data":"7140e724bc348d5015fa39a0f4c4b264b193f6e3f9740abb0d81322e22b66661"}
Nov 03 13:19:51.823072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.822706    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7140e724bc348d5015fa39a0f4c4b264b193f6e3f9740abb0d81322e22b66661"
Nov 03 13:19:51.970646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.970620    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:51.998966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:51.998931    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:52.078834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.078754    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-openshift-ca\") pod \"06cacf08-f842-4b04-b873-e55a01365aa5\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") "
Nov 03 13:19:52.078834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.078812    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-xks-ca\") pod \"06cacf08-f842-4b04-b873-e55a01365aa5\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") "
Nov 03 13:19:52.078834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.078836    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06cacf08-f842-4b04-b873-e55a01365aa5-kube-api-access-v7qx7\" (UniqueName: \"kubernetes.io/projected/06cacf08-f842-4b04-b873-e55a01365aa5-kube-api-access-v7qx7\") pod \"06cacf08-f842-4b04-b873-e55a01365aa5\" (UID: \"06cacf08-f842-4b04-b873-e55a01365aa5\") "
Nov 03 13:19:52.079137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.079092    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-openshift-ca" pod "06cacf08-f842-4b04-b873-e55a01365aa5" (UID: "06cacf08-f842-4b04-b873-e55a01365aa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:52.081052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.081023    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-xks-ca" pod "06cacf08-f842-4b04-b873-e55a01365aa5" (UID: "06cacf08-f842-4b04-b873-e55a01365aa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:52.081190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.081151    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cacf08-f842-4b04-b873-e55a01365aa5-kube-api-access-v7qx7" pod "06cacf08-f842-4b04-b873-e55a01365aa5" (UID: "06cacf08-f842-4b04-b873-e55a01365aa5"). InnerVolumeSpecName "kube-api-access-v7qx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:52.179806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.179775    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:52.179806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.179801    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cacf08-f842-4b04-b873-e55a01365aa5-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:52.179806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.179811    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7qx7\" (UniqueName: \"kubernetes.io/projected/06cacf08-f842-4b04-b873-e55a01365aa5-kube-api-access-v7qx7\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:52.309671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.309638    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:52.702837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.702802    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:52.830248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.830200    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg" event={"ID":"06cacf08-f842-4b04-b873-e55a01365aa5","Type":"ContainerDied","Data":"a49eddd7eb9d189e3295933df0f8c6c0bf86b53c36961bde19aaf14d3fa72e2b"}
Nov 03 13:19:52.830628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.830262    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"
Nov 03 13:19:52.830628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:52.830289    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a49eddd7eb9d189e3295933df0f8c6c0bf86b53c36961bde19aaf14d3fa72e2b"
Nov 03 13:19:53.318397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:53.318360    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:53.912926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:53.912881    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:54.597495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:54.597457    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:54.731544 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:19:54.731510    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:19:56.387644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:56.387559    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:56.529781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:56.529749    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:56.798693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:56.798660    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:57.385983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:57.385896    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:58.016116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:58.016066    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:58.547848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:58.547814    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:19:58.713337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:58.713304    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:59.491550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:19:59.491515    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:00.183360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:00.183326    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:01.072202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.072166    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"]
Nov 03 13:20:01.072640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.072627    5420 cpu_manager.go:414] "containerMap: removing container" podUID="06cacf08-f842-4b04-b873-e55a01365aa5" containerName="revoke-keys"
Nov 03 13:20:01.072678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.072641    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cacf08-f842-4b04-b873-e55a01365aa5" containerName="revoke-keys"
Nov 03 13:20:01.072678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.072659    5420 cpu_manager.go:414] "containerMap: removing container" podUID="23e481f0-0b30-4881-87ac-e0e96a798411" containerName="revoke-keys"
Nov 03 13:20:01.072678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.072664    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e481f0-0b30-4881-87ac-e0e96a798411" containerName="revoke-keys"
Nov 03 13:20:01.072795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.072751    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cacf08-f842-4b04-b873-e55a01365aa5" containerName="revoke-keys"
Nov 03 13:20:01.072795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.072761    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e481f0-0b30-4881-87ac-e0e96a798411" containerName="revoke-keys"
Nov 03 13:20:01.075980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.075961    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.080432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.080408    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:20:01.081540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.081519    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:20:01.081674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.081592    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:20:01.081674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.081593    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:20:01.093203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.087228    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"]
Nov 03 13:20:01.195537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.195496    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck56g\" (UniqueName: \"kubernetes.io/projected/e726ad73-dc4c-4265-aa67-30094dd2c4e9-kube-api-access-ck56g\") pod \"maas-api-revoke-sub-7474504f-d7zr8\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.195724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.195573    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7474504f-d7zr8\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.195724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.195689    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7474504f-d7zr8\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.222514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.222478    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:01.297299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.297256    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck56g\" (UniqueName: \"kubernetes.io/projected/e726ad73-dc4c-4265-aa67-30094dd2c4e9-kube-api-access-ck56g\") pod \"maas-api-revoke-sub-7474504f-d7zr8\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.297495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.297345    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7474504f-d7zr8\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.297495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.297390    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7474504f-d7zr8\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.298122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.298095    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7474504f-d7zr8\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.299775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.299742    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7474504f-d7zr8\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.307625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.307600    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck56g\" (UniqueName: \"kubernetes.io/projected/e726ad73-dc4c-4265-aa67-30094dd2c4e9-kube-api-access-ck56g\") pod \"maas-api-revoke-sub-7474504f-d7zr8\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.403010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.402928    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:01.682291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.682260    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"]
Nov 03 13:20:01.683741 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:01.683717    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode726ad73_dc4c_4265_aa67_30094dd2c4e9.slice/crio-b4196883bc72ef9b546546b1fb44b5d01fc356d103d4f0c1938f85fe1bcedf31 WatchSource:0}: Error finding container b4196883bc72ef9b546546b1fb44b5d01fc356d103d4f0c1938f85fe1bcedf31: Status 404 returned error can't find the container with id b4196883bc72ef9b546546b1fb44b5d01fc356d103d4f0c1938f85fe1bcedf31
Nov 03 13:20:01.886431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:01.886397    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8" event={"ID":"e726ad73-dc4c-4265-aa67-30094dd2c4e9","Type":"ContainerStarted","Data":"b4196883bc72ef9b546546b1fb44b5d01fc356d103d4f0c1938f85fe1bcedf31"}
Nov 03 13:20:02.006282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.006206    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:02.222658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.222622    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"]
Nov 03 13:20:02.226499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.226481    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.233197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.233168    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"]
Nov 03 13:20:02.411026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.410988    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgzj\" (UniqueName: \"kubernetes.io/projected/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-kube-api-access-jcgzj\") pod \"maas-api-revoke-sub-96a4069b-kkp5f\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.411026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.411034    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a4069b-kkp5f\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.411353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.411199    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a4069b-kkp5f\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.512200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.512112    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgzj\" (UniqueName: \"kubernetes.io/projected/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-kube-api-access-jcgzj\") pod \"maas-api-revoke-sub-96a4069b-kkp5f\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.512200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.512147    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a4069b-kkp5f\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.512200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.512198    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a4069b-kkp5f\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.512834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.512808    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a4069b-kkp5f\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.514655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.514636    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a4069b-kkp5f\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.521411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.521389    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgzj\" (UniqueName: \"kubernetes.io/projected/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-kube-api-access-jcgzj\") pod \"maas-api-revoke-sub-96a4069b-kkp5f\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.543254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.543232    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:02.758235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.758179    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:02.821369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.821339    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"]
Nov 03 13:20:02.822504 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:02.822428    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9b06ea_6c01_40f0_b0ee_0fae07ac7d97.slice/crio-89cae3e3abf4e629406e3406aeb952da461d0260e0c2636a0552b9944bd0f499 WatchSource:0}: Error finding container 89cae3e3abf4e629406e3406aeb952da461d0260e0c2636a0552b9944bd0f499: Status 404 returned error can't find the container with id 89cae3e3abf4e629406e3406aeb952da461d0260e0c2636a0552b9944bd0f499
Nov 03 13:20:02.893877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.893848    5420 generic.go:356] "Generic (PLEG): container finished" podID="e726ad73-dc4c-4265-aa67-30094dd2c4e9" containerID="e3f7ee5f1c315dd05971796638a798a300526e541174ce4868d0daa62012c50c" exitCode=0
Nov 03 13:20:02.894062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.893937    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8" event={"ID":"e726ad73-dc4c-4265-aa67-30094dd2c4e9","Type":"ContainerDied","Data":"e3f7ee5f1c315dd05971796638a798a300526e541174ce4868d0daa62012c50c"}
Nov 03 13:20:02.895299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:02.895275    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f" event={"ID":"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97","Type":"ContainerStarted","Data":"89cae3e3abf4e629406e3406aeb952da461d0260e0c2636a0552b9944bd0f499"}
Nov 03 13:20:03.613581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:03.613537    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:03.902792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:03.902707    5420 generic.go:356] "Generic (PLEG): container finished" podID="ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97" containerID="af08b388fe7ab32b80a6270b40a8493306d780f50053684a7fb8ed3e1b63d6fc" exitCode=0
Nov 03 13:20:03.902951 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:03.902796    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f" event={"ID":"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97","Type":"ContainerDied","Data":"af08b388fe7ab32b80a6270b40a8493306d780f50053684a7fb8ed3e1b63d6fc"}
Nov 03 13:20:04.034290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.034267    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:04.128231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.128174    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-xks-ca\") pod \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") "
Nov 03 13:20:04.128399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.128255    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e726ad73-dc4c-4265-aa67-30094dd2c4e9-kube-api-access-ck56g\" (UniqueName: \"kubernetes.io/projected/e726ad73-dc4c-4265-aa67-30094dd2c4e9-kube-api-access-ck56g\") pod \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") "
Nov 03 13:20:04.128399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.128302    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-openshift-ca\") pod \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\" (UID: \"e726ad73-dc4c-4265-aa67-30094dd2c4e9\") "
Nov 03 13:20:04.128746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.128720    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-openshift-ca" pod "e726ad73-dc4c-4265-aa67-30094dd2c4e9" (UID: "e726ad73-dc4c-4265-aa67-30094dd2c4e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:04.130236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.130195    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-xks-ca" pod "e726ad73-dc4c-4265-aa67-30094dd2c4e9" (UID: "e726ad73-dc4c-4265-aa67-30094dd2c4e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:04.130379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.130356    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e726ad73-dc4c-4265-aa67-30094dd2c4e9-kube-api-access-ck56g" pod "e726ad73-dc4c-4265-aa67-30094dd2c4e9" (UID: "e726ad73-dc4c-4265-aa67-30094dd2c4e9"). InnerVolumeSpecName "kube-api-access-ck56g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:04.230004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.229971    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:04.230004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.230001    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck56g\" (UniqueName: \"kubernetes.io/projected/e726ad73-dc4c-4265-aa67-30094dd2c4e9-kube-api-access-ck56g\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:04.230199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.230012    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e726ad73-dc4c-4265-aa67-30094dd2c4e9-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:04.403383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.403333    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:04.910086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.910050    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"
Nov 03 13:20:04.910086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.910073    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8" event={"ID":"e726ad73-dc4c-4265-aa67-30094dd2c4e9","Type":"ContainerDied","Data":"b4196883bc72ef9b546546b1fb44b5d01fc356d103d4f0c1938f85fe1bcedf31"}
Nov 03 13:20:04.910577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:04.910159    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4196883bc72ef9b546546b1fb44b5d01fc356d103d4f0c1938f85fe1bcedf31"
Nov 03 13:20:05.044584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.044557    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:05.140149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.140111    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-openshift-ca\") pod \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") "
Nov 03 13:20:05.140383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.140367    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-kube-api-access-jcgzj\" (UniqueName: \"kubernetes.io/projected/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-kube-api-access-jcgzj\") pod \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") "
Nov 03 13:20:05.141392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.140568    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-openshift-ca" pod "ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97" (UID: "ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:05.141392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.140466    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-xks-ca\") pod \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\" (UID: \"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97\") "
Nov 03 13:20:05.141392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.141101    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:05.141579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.141392    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:05.142853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.142828    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-kube-api-access-jcgzj" pod "ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97" (UID: "ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97"). InnerVolumeSpecName "kube-api-access-jcgzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:05.143037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.143012    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-xks-ca" pod "ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97" (UID: "ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:05.242603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.242573    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcgzj\" (UniqueName: \"kubernetes.io/projected/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-kube-api-access-jcgzj\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:05.242603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.242601    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:05.918083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.918054    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"
Nov 03 13:20:05.918083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.918069    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f" event={"ID":"ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97","Type":"ContainerDied","Data":"89cae3e3abf4e629406e3406aeb952da461d0260e0c2636a0552b9944bd0f499"}
Nov 03 13:20:05.918522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:05.918127    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89cae3e3abf4e629406e3406aeb952da461d0260e0c2636a0552b9944bd0f499"
Nov 03 13:20:06.024115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:06.020645    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:06.850232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:06.850150    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:07.734119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:07.734085    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:08.509430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:08.509394    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:09.163802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:09.163763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:09.803880 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:09.803841    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:10.110872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.110801    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"]
Nov 03 13:20:10.111275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.111262    5420 cpu_manager.go:414] "containerMap: removing container" podUID="ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97" containerName="revoke-keys"
Nov 03 13:20:10.111326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.111276    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97" containerName="revoke-keys"
Nov 03 13:20:10.111326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.111287    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e726ad73-dc4c-4265-aa67-30094dd2c4e9" containerName="revoke-keys"
Nov 03 13:20:10.111326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.111293    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e726ad73-dc4c-4265-aa67-30094dd2c4e9" containerName="revoke-keys"
Nov 03 13:20:10.111411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.111386    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97" containerName="revoke-keys"
Nov 03 13:20:10.111411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.111394    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e726ad73-dc4c-4265-aa67-30094dd2c4e9" containerName="revoke-keys"
Nov 03 13:20:10.115577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.115560    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.119747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.119725    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:20:10.121192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.121169    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:20:10.121322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.121237    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:20:10.121322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.121235    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:20:10.122094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.122069    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"]
Nov 03 13:20:10.203226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.203181    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9lr5\" (UniqueName: \"kubernetes.io/projected/5e3369c1-1168-45e1-838c-5e8e9645dd3c-kube-api-access-m9lr5\") pod \"maas-api-revoke-sub-1c3f5c81-2qpvn\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.203609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.203250    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c3f5c81-2qpvn\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.203609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.203335    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c3f5c81-2qpvn\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.304643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.304609    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c3f5c81-2qpvn\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.304821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.304657    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c3f5c81-2qpvn\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.304821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.304801    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9lr5\" (UniqueName: \"kubernetes.io/projected/5e3369c1-1168-45e1-838c-5e8e9645dd3c-kube-api-access-m9lr5\") pod \"maas-api-revoke-sub-1c3f5c81-2qpvn\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.305290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.305261    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c3f5c81-2qpvn\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.306903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.306880    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c3f5c81-2qpvn\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.313785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.313762    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9lr5\" (UniqueName: \"kubernetes.io/projected/5e3369c1-1168-45e1-838c-5e8e9645dd3c-kube-api-access-m9lr5\") pod \"maas-api-revoke-sub-1c3f5c81-2qpvn\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.431501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.431426    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:10.473957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.473920    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:10.714172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.714146    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"]
Nov 03 13:20:10.715463 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:10.715428    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3369c1_1168_45e1_838c_5e8e9645dd3c.slice/crio-5aa895daea61f6b389eb352e7636234b6aa84fefba56b24546da1e8035c05a5f WatchSource:0}: Error finding container 5aa895daea61f6b389eb352e7636234b6aa84fefba56b24546da1e8035c05a5f: Status 404 returned error can't find the container with id 5aa895daea61f6b389eb352e7636234b6aa84fefba56b24546da1e8035c05a5f
Nov 03 13:20:10.949746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:10.949712    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn" event={"ID":"5e3369c1-1168-45e1-838c-5e8e9645dd3c","Type":"ContainerStarted","Data":"5aa895daea61f6b389eb352e7636234b6aa84fefba56b24546da1e8035c05a5f"}
Nov 03 13:20:11.106402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:11.106362    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:11.693427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:11.693393    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:11.957434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:11.957406    5420 generic.go:356] "Generic (PLEG): container finished" podID="5e3369c1-1168-45e1-838c-5e8e9645dd3c" containerID="c2e1249ff41a131e04974b1211eea5085252d0b531705a2556411d35cba57f16" exitCode=0
Nov 03 13:20:11.957618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:11.957491    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn" event={"ID":"5e3369c1-1168-45e1-838c-5e8e9645dd3c","Type":"ContainerDied","Data":"c2e1249ff41a131e04974b1211eea5085252d0b531705a2556411d35cba57f16"}
Nov 03 13:20:12.314265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:12.314086    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:12.894191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:12.894156    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:13.110320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.110297    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:13.239584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.239554    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-openshift-ca\") pod \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") "
Nov 03 13:20:13.239768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.239638    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3369c1-1168-45e1-838c-5e8e9645dd3c-kube-api-access-m9lr5\" (UniqueName: \"kubernetes.io/projected/5e3369c1-1168-45e1-838c-5e8e9645dd3c-kube-api-access-m9lr5\") pod \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") "
Nov 03 13:20:13.239768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.239685    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-xks-ca\") pod \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\" (UID: \"5e3369c1-1168-45e1-838c-5e8e9645dd3c\") "
Nov 03 13:20:13.240009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.239973    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-openshift-ca" pod "5e3369c1-1168-45e1-838c-5e8e9645dd3c" (UID: "5e3369c1-1168-45e1-838c-5e8e9645dd3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:13.241798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.241777    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-xks-ca" pod "5e3369c1-1168-45e1-838c-5e8e9645dd3c" (UID: "5e3369c1-1168-45e1-838c-5e8e9645dd3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:13.241860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.241798    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3369c1-1168-45e1-838c-5e8e9645dd3c-kube-api-access-m9lr5" pod "5e3369c1-1168-45e1-838c-5e8e9645dd3c" (UID: "5e3369c1-1168-45e1-838c-5e8e9645dd3c"). InnerVolumeSpecName "kube-api-access-m9lr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:13.340894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.340861    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9lr5\" (UniqueName: \"kubernetes.io/projected/5e3369c1-1168-45e1-838c-5e8e9645dd3c-kube-api-access-m9lr5\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:13.340894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.340892    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:13.341094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.340909    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3369c1-1168-45e1-838c-5e8e9645dd3c-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:13.595462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.595359    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:13.866892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.866816    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"]
Nov 03 13:20:13.867326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.867311    5420 cpu_manager.go:414] "containerMap: removing container" podUID="5e3369c1-1168-45e1-838c-5e8e9645dd3c" containerName="revoke-keys"
Nov 03 13:20:13.867367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.867328    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3369c1-1168-45e1-838c-5e8e9645dd3c" containerName="revoke-keys"
Nov 03 13:20:13.867422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.867413    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3369c1-1168-45e1-838c-5e8e9645dd3c" containerName="revoke-keys"
Nov 03 13:20:13.870729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.870705    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:13.877336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.877312    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"]
Nov 03 13:20:13.947589 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.947553    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0506a5e3-dgsx5\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:13.948003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.947685    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0506a5e3-dgsx5\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:13.948003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.947738    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtf5\" (UniqueName: \"kubernetes.io/projected/b69242a8-f26c-421b-a78c-cae43aaf4f25-kube-api-access-hdtf5\") pod \"maas-api-revoke-sub-0506a5e3-dgsx5\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:13.971946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.971917    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"
Nov 03 13:20:13.971946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.971925    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn" event={"ID":"5e3369c1-1168-45e1-838c-5e8e9645dd3c","Type":"ContainerDied","Data":"5aa895daea61f6b389eb352e7636234b6aa84fefba56b24546da1e8035c05a5f"}
Nov 03 13:20:13.972160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:13.971985    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aa895daea61f6b389eb352e7636234b6aa84fefba56b24546da1e8035c05a5f"
Nov 03 13:20:14.049033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.049000    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0506a5e3-dgsx5\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:14.049283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.049087    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0506a5e3-dgsx5\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:14.049283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.049135    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtf5\" (UniqueName: \"kubernetes.io/projected/b69242a8-f26c-421b-a78c-cae43aaf4f25-kube-api-access-hdtf5\") pod \"maas-api-revoke-sub-0506a5e3-dgsx5\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:14.049815 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.049792    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0506a5e3-dgsx5\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:14.051477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.051460    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0506a5e3-dgsx5\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:14.057429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.057411    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtf5\" (UniqueName: \"kubernetes.io/projected/b69242a8-f26c-421b-a78c-cae43aaf4f25-kube-api-access-hdtf5\") pod \"maas-api-revoke-sub-0506a5e3-dgsx5\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:14.185056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.184965    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:14.217486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.217453    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:14.339165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.339131    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:20:14.474987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.474964    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"]
Nov 03 13:20:14.476619 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:14.476592    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69242a8_f26c_421b_a78c_cae43aaf4f25.slice/crio-132f59a6dc8be445c2256456a47a38363291d84ed99528f32b476267254e3007 WatchSource:0}: Error finding container 132f59a6dc8be445c2256456a47a38363291d84ed99528f32b476267254e3007: Status 404 returned error can't find the container with id 132f59a6dc8be445c2256456a47a38363291d84ed99528f32b476267254e3007
Nov 03 13:20:14.582477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.582447    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"]
Nov 03 13:20:14.587499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.587481    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.597974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.597546    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"]
Nov 03 13:20:14.656005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.655960    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-259e5448-p5xzk\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.656149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.656130    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-259e5448-p5xzk\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.656280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.656261    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dfbs\" (UniqueName: \"kubernetes.io/projected/9c449edd-6bfc-4903-80b3-1b0c09b1d158-kube-api-access-4dfbs\") pod \"maas-api-revoke-sub-259e5448-p5xzk\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.757819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.757789    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dfbs\" (UniqueName: \"kubernetes.io/projected/9c449edd-6bfc-4903-80b3-1b0c09b1d158-kube-api-access-4dfbs\") pod \"maas-api-revoke-sub-259e5448-p5xzk\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.757941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.757876    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-259e5448-p5xzk\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.757985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.757941    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-259e5448-p5xzk\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.758712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.758683    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-259e5448-p5xzk\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.760251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.760223    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-259e5448-p5xzk\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.767145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.767119    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dfbs\" (UniqueName: \"kubernetes.io/projected/9c449edd-6bfc-4903-80b3-1b0c09b1d158-kube-api-access-4dfbs\") pod \"maas-api-revoke-sub-259e5448-p5xzk\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.904953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.904917    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:14.922501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.922466    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:15.006073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:14.999967    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5" event={"ID":"b69242a8-f26c-421b-a78c-cae43aaf4f25","Type":"ContainerStarted","Data":"daf4d0cbf08dd9d1d88cd119b82f5210ed8fb19612d9eac10b0bd076585cacd9"}
Nov 03 13:20:15.006073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:15.000037    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5" event={"ID":"b69242a8-f26c-421b-a78c-cae43aaf4f25","Type":"ContainerStarted","Data":"132f59a6dc8be445c2256456a47a38363291d84ed99528f32b476267254e3007"}
Nov 03 13:20:15.039313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:15.037137    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5" podStartSLOduration=2.037114012 podStartE2EDuration="2.037114012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:15.032528657 +0000 UTC m=+656.110293685" watchObservedRunningTime="2026-11-03 13:20:15.037114012 +0000 UTC m=+656.114879035"
Nov 03 13:20:15.199969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:15.199938    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"]
Nov 03 13:20:15.201950 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:15.201917    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c449edd_6bfc_4903_80b3_1b0c09b1d158.slice/crio-85d2cfc3c6cedc03ca0ee00e93b2eac0964b66bf2675abc0fabaaa8a326427bd WatchSource:0}: Error finding container 85d2cfc3c6cedc03ca0ee00e93b2eac0964b66bf2675abc0fabaaa8a326427bd: Status 404 returned error can't find the container with id 85d2cfc3c6cedc03ca0ee00e93b2eac0964b66bf2675abc0fabaaa8a326427bd
Nov 03 13:20:15.386252 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:20:15.385067    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69242a8_f26c_421b_a78c_cae43aaf4f25.slice/crio-conmon-daf4d0cbf08dd9d1d88cd119b82f5210ed8fb19612d9eac10b0bd076585cacd9.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:20:15.386252 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:20:15.385377    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69242a8_f26c_421b_a78c_cae43aaf4f25.slice/crio-conmon-daf4d0cbf08dd9d1d88cd119b82f5210ed8fb19612d9eac10b0bd076585cacd9.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:20:15.584100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:15.584022    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:16.007531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:16.007495    5420 generic.go:356] "Generic (PLEG): container finished" podID="b69242a8-f26c-421b-a78c-cae43aaf4f25" containerID="daf4d0cbf08dd9d1d88cd119b82f5210ed8fb19612d9eac10b0bd076585cacd9" exitCode=0
Nov 03 13:20:16.007925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:16.007574    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5" event={"ID":"b69242a8-f26c-421b-a78c-cae43aaf4f25","Type":"ContainerDied","Data":"daf4d0cbf08dd9d1d88cd119b82f5210ed8fb19612d9eac10b0bd076585cacd9"}
Nov 03 13:20:16.009180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:16.009159    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk" event={"ID":"9c449edd-6bfc-4903-80b3-1b0c09b1d158","Type":"ContainerStarted","Data":"d10d7cc1ff97d04cea21c8e93efa1f4171ca15c2f9745ab5eaa8aa324b57da3f"}
Nov 03 13:20:16.009306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:16.009200    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk" event={"ID":"9c449edd-6bfc-4903-80b3-1b0c09b1d158","Type":"ContainerStarted","Data":"85d2cfc3c6cedc03ca0ee00e93b2eac0964b66bf2675abc0fabaaa8a326427bd"}
Nov 03 13:20:16.043288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:16.043236    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk" podStartSLOduration=2.043219982 podStartE2EDuration="2.043219982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:16.040566139 +0000 UTC m=+657.118331167" watchObservedRunningTime="2026-11-03 13:20:16.043219982 +0000 UTC m=+657.120985000"
Nov 03 13:20:16.208701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:16.208663    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:16.904904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:16.904869    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:17.016912 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.016875    5420 generic.go:356] "Generic (PLEG): container finished" podID="9c449edd-6bfc-4903-80b3-1b0c09b1d158" containerID="d10d7cc1ff97d04cea21c8e93efa1f4171ca15c2f9745ab5eaa8aa324b57da3f" exitCode=0
Nov 03 13:20:17.017382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.016957    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk" event={"ID":"9c449edd-6bfc-4903-80b3-1b0c09b1d158","Type":"ContainerDied","Data":"d10d7cc1ff97d04cea21c8e93efa1f4171ca15c2f9745ab5eaa8aa324b57da3f"}
Nov 03 13:20:17.050449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.050417    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:17.164870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.164807    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:17.297941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.297904    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-openshift-ca\") pod \"b69242a8-f26c-421b-a78c-cae43aaf4f25\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") "
Nov 03 13:20:17.298143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.298086    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-xks-ca\") pod \"b69242a8-f26c-421b-a78c-cae43aaf4f25\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") "
Nov 03 13:20:17.298143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.298135    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69242a8-f26c-421b-a78c-cae43aaf4f25-kube-api-access-hdtf5\" (UniqueName: \"kubernetes.io/projected/b69242a8-f26c-421b-a78c-cae43aaf4f25-kube-api-access-hdtf5\") pod \"b69242a8-f26c-421b-a78c-cae43aaf4f25\" (UID: \"b69242a8-f26c-421b-a78c-cae43aaf4f25\") "
Nov 03 13:20:17.298372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.298342    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-openshift-ca" pod "b69242a8-f26c-421b-a78c-cae43aaf4f25" (UID: "b69242a8-f26c-421b-a78c-cae43aaf4f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:17.298570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.298552    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:17.300307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.300275    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69242a8-f26c-421b-a78c-cae43aaf4f25-kube-api-access-hdtf5" pod "b69242a8-f26c-421b-a78c-cae43aaf4f25" (UID: "b69242a8-f26c-421b-a78c-cae43aaf4f25"). InnerVolumeSpecName "kube-api-access-hdtf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:17.300307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.300290    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-xks-ca" pod "b69242a8-f26c-421b-a78c-cae43aaf4f25" (UID: "b69242a8-f26c-421b-a78c-cae43aaf4f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:17.400166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.400134    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69242a8-f26c-421b-a78c-cae43aaf4f25-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:17.400166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.400161    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdtf5\" (UniqueName: \"kubernetes.io/projected/b69242a8-f26c-421b-a78c-cae43aaf4f25-kube-api-access-hdtf5\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:17.519978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:17.519924    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:18.023870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.023842    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"
Nov 03 13:20:18.024418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.023876    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5" event={"ID":"b69242a8-f26c-421b-a78c-cae43aaf4f25","Type":"ContainerDied","Data":"132f59a6dc8be445c2256456a47a38363291d84ed99528f32b476267254e3007"}
Nov 03 13:20:18.024418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.023935    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="132f59a6dc8be445c2256456a47a38363291d84ed99528f32b476267254e3007"
Nov 03 13:20:18.182106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.182084    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:18.233894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.233850    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:18.310852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.310757    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-xks-ca\") pod \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") "
Nov 03 13:20:18.311016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.310880    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-openshift-ca\") pod \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") "
Nov 03 13:20:18.311074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.311035    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c449edd-6bfc-4903-80b3-1b0c09b1d158-kube-api-access-4dfbs\" (UniqueName: \"kubernetes.io/projected/9c449edd-6bfc-4903-80b3-1b0c09b1d158-kube-api-access-4dfbs\") pod \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\" (UID: \"9c449edd-6bfc-4903-80b3-1b0c09b1d158\") "
Nov 03 13:20:18.311319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.311285    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-openshift-ca" pod "9c449edd-6bfc-4903-80b3-1b0c09b1d158" (UID: "9c449edd-6bfc-4903-80b3-1b0c09b1d158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:18.311495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.311472    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:18.312966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.312932    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-xks-ca" pod "9c449edd-6bfc-4903-80b3-1b0c09b1d158" (UID: "9c449edd-6bfc-4903-80b3-1b0c09b1d158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:18.313055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.313036    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c449edd-6bfc-4903-80b3-1b0c09b1d158-kube-api-access-4dfbs" pod "9c449edd-6bfc-4903-80b3-1b0c09b1d158" (UID: "9c449edd-6bfc-4903-80b3-1b0c09b1d158"). InnerVolumeSpecName "kube-api-access-4dfbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:18.412799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.412769    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c449edd-6bfc-4903-80b3-1b0c09b1d158-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:18.412799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.412797    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dfbs\" (UniqueName: \"kubernetes.io/projected/9c449edd-6bfc-4903-80b3-1b0c09b1d158-kube-api-access-4dfbs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:18.938878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:18.938847    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:19.032681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:19.032642    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk" event={"ID":"9c449edd-6bfc-4903-80b3-1b0c09b1d158","Type":"ContainerDied","Data":"85d2cfc3c6cedc03ca0ee00e93b2eac0964b66bf2675abc0fabaaa8a326427bd"}
Nov 03 13:20:19.033061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:19.032735    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d2cfc3c6cedc03ca0ee00e93b2eac0964b66bf2675abc0fabaaa8a326427bd"
Nov 03 13:20:19.033061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:19.032652    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"
Nov 03 13:20:19.624128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:19.624074    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:20.382428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:20.382394    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:21.196192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:21.196162    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:22.181560 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:22.181519    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:23.023594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:23.023555    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:23.605146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:23.605109    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:24.218289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.218248    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:24.732434 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:20:24.732395    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:20:24.778275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.778243    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"]
Nov 03 13:20:24.778708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.778696    5420 cpu_manager.go:414] "containerMap: removing container" podUID="9c449edd-6bfc-4903-80b3-1b0c09b1d158" containerName="revoke-keys"
Nov 03 13:20:24.778755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.778709    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c449edd-6bfc-4903-80b3-1b0c09b1d158" containerName="revoke-keys"
Nov 03 13:20:24.778755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.778740    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b69242a8-f26c-421b-a78c-cae43aaf4f25" containerName="revoke-keys"
Nov 03 13:20:24.778755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.778745    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69242a8-f26c-421b-a78c-cae43aaf4f25" containerName="revoke-keys"
Nov 03 13:20:24.778841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.778818    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69242a8-f26c-421b-a78c-cae43aaf4f25" containerName="revoke-keys"
Nov 03 13:20:24.778841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.778830    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c449edd-6bfc-4903-80b3-1b0c09b1d158" containerName="revoke-keys"
Nov 03 13:20:24.783633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.783613    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:24.788059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.788033    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:20:24.788272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.788111    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:20:24.788272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.788125    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:20:24.788272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.788142    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:20:24.790798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.790770    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"]
Nov 03 13:20:24.895742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.895709    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a895c6-z9bkw\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:24.895937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.895856    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a895c6-z9bkw\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:24.895937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.895891    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6927\" (UniqueName: \"kubernetes.io/projected/4a894dde-efd1-488a-937b-5f69b9905c9d-kube-api-access-h6927\") pod \"maas-api-revoke-sub-f0a895c6-z9bkw\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:24.997147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.997061    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a895c6-z9bkw\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:24.997361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.997157    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a895c6-z9bkw\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:24.997361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.997183    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6927\" (UniqueName: \"kubernetes.io/projected/4a894dde-efd1-488a-937b-5f69b9905c9d-kube-api-access-h6927\") pod \"maas-api-revoke-sub-f0a895c6-z9bkw\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:24.997830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.997809    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a895c6-z9bkw\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:24.999547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:24.999529    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a895c6-z9bkw\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:25.008036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:25.008012    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6927\" (UniqueName: \"kubernetes.io/projected/4a894dde-efd1-488a-937b-5f69b9905c9d-kube-api-access-h6927\") pod \"maas-api-revoke-sub-f0a895c6-z9bkw\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:25.102297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:25.102265    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:25.331288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:25.331195    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:20:25.380014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:25.379971    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"]
Nov 03 13:20:25.383412 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:25.383386    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a894dde_efd1_488a_937b_5f69b9905c9d.slice/crio-d5c3f6a7dab25c28fc9c0c8faefc0ec8e311300f7c0d3160f250ca522ec62994 WatchSource:0}: Error finding container d5c3f6a7dab25c28fc9c0c8faefc0ec8e311300f7c0d3160f250ca522ec62994: Status 404 returned error can't find the container with id d5c3f6a7dab25c28fc9c0c8faefc0ec8e311300f7c0d3160f250ca522ec62994
Nov 03 13:20:25.954004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:25.953944    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:26.079683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:26.079591    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw" event={"ID":"4a894dde-efd1-488a-937b-5f69b9905c9d","Type":"ContainerStarted","Data":"08dd67dc1ff42757087f318069db624140b7b338c371876d9dab4c7322179a9e"}
Nov 03 13:20:26.079683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:26.079651    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw" event={"ID":"4a894dde-efd1-488a-937b-5f69b9905c9d","Type":"ContainerStarted","Data":"d5c3f6a7dab25c28fc9c0c8faefc0ec8e311300f7c0d3160f250ca522ec62994"}
Nov 03 13:20:26.132695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:26.132622    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw" podStartSLOduration=2.132596812 podStartE2EDuration="2.132596812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:26.100627662 +0000 UTC m=+667.178392696" watchObservedRunningTime="2026-11-03 13:20:26.132596812 +0000 UTC m=+667.210361838"
Nov 03 13:20:26.144252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:26.144170    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:26.291090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:26.290985    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:27.087081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.087048    5420 generic.go:356] "Generic (PLEG): container finished" podID="4a894dde-efd1-488a-937b-5f69b9905c9d" containerID="08dd67dc1ff42757087f318069db624140b7b338c371876d9dab4c7322179a9e" exitCode=0
Nov 03 13:20:27.087538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.087130    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw" event={"ID":"4a894dde-efd1-488a-937b-5f69b9905c9d","Type":"ContainerDied","Data":"08dd67dc1ff42757087f318069db624140b7b338c371876d9dab4c7322179a9e"}
Nov 03 13:20:27.099548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.099522    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:27.729149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.729117    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"]
Nov 03 13:20:27.733011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.732994    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:27.740261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.740233    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"]
Nov 03 13:20:27.757860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.757829    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29189215-57mr9\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:27.758015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.757953    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29189215-57mr9\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:27.758066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.758055    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqh6r\" (UniqueName: \"kubernetes.io/projected/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-kube-api-access-cqh6r\") pod \"maas-api-revoke-sub-29189215-57mr9\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:27.791161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.791059    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:27.859322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.859287    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29189215-57mr9\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:27.859499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.859367    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29189215-57mr9\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:27.859499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.859460    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqh6r\" (UniqueName: \"kubernetes.io/projected/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-kube-api-access-cqh6r\") pod \"maas-api-revoke-sub-29189215-57mr9\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:27.860076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.860050    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29189215-57mr9\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:27.861831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.861807    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29189215-57mr9\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:27.867519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:27.867501    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqh6r\" (UniqueName: \"kubernetes.io/projected/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-kube-api-access-cqh6r\") pod \"maas-api-revoke-sub-29189215-57mr9\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:28.049607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.049521    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:28.252996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.252972    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:28.264080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.264057    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-openshift-ca\") pod \"4a894dde-efd1-488a-937b-5f69b9905c9d\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") "
Nov 03 13:20:28.264185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.264168    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-xks-ca\") pod \"4a894dde-efd1-488a-937b-5f69b9905c9d\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") "
Nov 03 13:20:28.264268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.264238    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a894dde-efd1-488a-937b-5f69b9905c9d-kube-api-access-h6927\" (UniqueName: \"kubernetes.io/projected/4a894dde-efd1-488a-937b-5f69b9905c9d-kube-api-access-h6927\") pod \"4a894dde-efd1-488a-937b-5f69b9905c9d\" (UID: \"4a894dde-efd1-488a-937b-5f69b9905c9d\") "
Nov 03 13:20:28.264389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.264368    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-openshift-ca" pod "4a894dde-efd1-488a-937b-5f69b9905c9d" (UID: "4a894dde-efd1-488a-937b-5f69b9905c9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:28.264578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.264561    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:28.266257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.266230    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a894dde-efd1-488a-937b-5f69b9905c9d-kube-api-access-h6927" pod "4a894dde-efd1-488a-937b-5f69b9905c9d" (UID: "4a894dde-efd1-488a-937b-5f69b9905c9d"). InnerVolumeSpecName "kube-api-access-h6927". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:28.266345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.266279    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-xks-ca" pod "4a894dde-efd1-488a-937b-5f69b9905c9d" (UID: "4a894dde-efd1-488a-937b-5f69b9905c9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:28.330978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.330953    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"]
Nov 03 13:20:28.332432 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:28.332398    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedd069f_adbf_4c74_94d8_ac5782e5e1c2.slice/crio-70a390f489d34d729844202fd2dfeefc66c35ce672430cc8af3bb74ca1bb2d4a WatchSource:0}: Error finding container 70a390f489d34d729844202fd2dfeefc66c35ce672430cc8af3bb74ca1bb2d4a: Status 404 returned error can't find the container with id 70a390f489d34d729844202fd2dfeefc66c35ce672430cc8af3bb74ca1bb2d4a
Nov 03 13:20:28.366040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.366013    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6927\" (UniqueName: \"kubernetes.io/projected/4a894dde-efd1-488a-937b-5f69b9905c9d-kube-api-access-h6927\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:28.366040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.366039    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894dde-efd1-488a-937b-5f69b9905c9d-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:28.506962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:28.506928    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:29.103404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:29.103367    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9" event={"ID":"bedd069f-adbf-4c74-94d8-ac5782e5e1c2","Type":"ContainerStarted","Data":"8260816a0f45e5bbbbbfc27518d4048c52d04f4d6f0cf9dfd1136f0797a03417"}
Nov 03 13:20:29.103580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:29.103419    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9" event={"ID":"bedd069f-adbf-4c74-94d8-ac5782e5e1c2","Type":"ContainerStarted","Data":"70a390f489d34d729844202fd2dfeefc66c35ce672430cc8af3bb74ca1bb2d4a"}
Nov 03 13:20:29.104761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:29.104735    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw" event={"ID":"4a894dde-efd1-488a-937b-5f69b9905c9d","Type":"ContainerDied","Data":"d5c3f6a7dab25c28fc9c0c8faefc0ec8e311300f7c0d3160f250ca522ec62994"}
Nov 03 13:20:29.104884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:29.104764    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:20:29.104884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:29.104775    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5c3f6a7dab25c28fc9c0c8faefc0ec8e311300f7c0d3160f250ca522ec62994"
Nov 03 13:20:29.126135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:29.126090    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9" podStartSLOduration=2.126077598 podStartE2EDuration="2.126077598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:29.122280402 +0000 UTC m=+670.200045428" watchObservedRunningTime="2026-11-03 13:20:29.126077598 +0000 UTC m=+670.203842649"
Nov 03 13:20:29.194420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:29.194381    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:29.813052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:29.813016    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:30.112523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:30.112434    5420 generic.go:356] "Generic (PLEG): container finished" podID="bedd069f-adbf-4c74-94d8-ac5782e5e1c2" containerID="8260816a0f45e5bbbbbfc27518d4048c52d04f4d6f0cf9dfd1136f0797a03417" exitCode=0
Nov 03 13:20:30.112523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:30.112477    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9" event={"ID":"bedd069f-adbf-4c74-94d8-ac5782e5e1c2","Type":"ContainerDied","Data":"8260816a0f45e5bbbbbfc27518d4048c52d04f4d6f0cf9dfd1136f0797a03417"}
Nov 03 13:20:30.393552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:30.393457    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:31.019234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.016737    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:31.260637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.260614    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:31.297666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.297577    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-kube-api-access-cqh6r\" (UniqueName: \"kubernetes.io/projected/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-kube-api-access-cqh6r\") pod \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") "
Nov 03 13:20:31.297830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.297662    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-xks-ca\") pod \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") "
Nov 03 13:20:31.297895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.297856    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-openshift-ca\") pod \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\" (UID: \"bedd069f-adbf-4c74-94d8-ac5782e5e1c2\") "
Nov 03 13:20:31.298282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.298239    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-openshift-ca" pod "bedd069f-adbf-4c74-94d8-ac5782e5e1c2" (UID: "bedd069f-adbf-4c74-94d8-ac5782e5e1c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:31.298418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.298394    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:31.300001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.299968    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-kube-api-access-cqh6r" pod "bedd069f-adbf-4c74-94d8-ac5782e5e1c2" (UID: "bedd069f-adbf-4c74-94d8-ac5782e5e1c2"). InnerVolumeSpecName "kube-api-access-cqh6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:31.300195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.300173    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-xks-ca" pod "bedd069f-adbf-4c74-94d8-ac5782e5e1c2" (UID: "bedd069f-adbf-4c74-94d8-ac5782e5e1c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:31.399043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.399007    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqh6r\" (UniqueName: \"kubernetes.io/projected/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-kube-api-access-cqh6r\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:31.399043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.399036    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedd069f-adbf-4c74-94d8-ac5782e5e1c2-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:31.411748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.411709    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"]
Nov 03 13:20:31.412307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.412284    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4a894dde-efd1-488a-937b-5f69b9905c9d" containerName="revoke-keys"
Nov 03 13:20:31.412307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.412307    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a894dde-efd1-488a-937b-5f69b9905c9d" containerName="revoke-keys"
Nov 03 13:20:31.412530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.412332    5420 cpu_manager.go:414] "containerMap: removing container" podUID="bedd069f-adbf-4c74-94d8-ac5782e5e1c2" containerName="revoke-keys"
Nov 03 13:20:31.412530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.412339    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedd069f-adbf-4c74-94d8-ac5782e5e1c2" containerName="revoke-keys"
Nov 03 13:20:31.412530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.412468    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedd069f-adbf-4c74-94d8-ac5782e5e1c2" containerName="revoke-keys"
Nov 03 13:20:31.412530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.412492    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a894dde-efd1-488a-937b-5f69b9905c9d" containerName="revoke-keys"
Nov 03 13:20:31.415904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.415883    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.432103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.432072    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"]
Nov 03 13:20:31.499070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.499042    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"]
Nov 03 13:20:31.500131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.500100    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.500272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.500169    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.500272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.500265    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.500382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.500346    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.500425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.500390    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.500538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.500515    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.500637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.500623    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhnm5\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-kube-api-access-nhnm5\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.500695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.500658    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.500750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.500709    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.502908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.502887    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.514940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.514914    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"]
Nov 03 13:20:31.601802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.601721    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhnm5\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-kube-api-access-nhnm5\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.601802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.601760    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.601802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.601780    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.601802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.601797    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.602080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.601866    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.602080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.601895    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de14b3f-b00e-4105-b5e9-4b52ef946958-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.602080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.601957    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.602080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.601981    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.602080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602010    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.602080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602062    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.602425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602156    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wkq\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-kube-api-access-k2wkq\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.602425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602194    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.602539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602427    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.602539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602447    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.602539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602530    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.602701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602571    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.602701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602611    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.602860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602701    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.602860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602752    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.602860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602789    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.602860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.602799    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.603116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.603093    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.603156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.603100    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.604708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.604688    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.605024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.605004    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.610655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.610636    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.610806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.610790    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhnm5\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-kube-api-access-nhnm5\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.687876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.687843    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:31.705414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.704402    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.704454    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de14b3f-b00e-4105-b5e9-4b52ef946958-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.704508    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.704551    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.704575    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wkq\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-kube-api-access-k2wkq\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.704611    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.704643    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.704673    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.704872    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.705706    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.705998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.705810    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.706741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.706703    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de14b3f-b00e-4105-b5e9-4b52ef946958-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.706965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.706944    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.707172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.707153    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.707886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.707858    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.709033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.709010    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.715202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.715180    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.715432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.715405    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wkq\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-kube-api-access-k2wkq\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:31.735333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.735293    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:31.822519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:31.822491    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:32.018825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.018802    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"]
Nov 03 13:20:32.019879 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:32.019856    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3beb4bb_7c9f_4b98_a1fd_4457bd6225a1.slice/crio-ae11ed494a894ccab9fb9ee3d92b00eb869c23ce019f824985131cc60c570568 WatchSource:0}: Error finding container ae11ed494a894ccab9fb9ee3d92b00eb869c23ce019f824985131cc60c570568: Status 404 returned error can't find the container with id ae11ed494a894ccab9fb9ee3d92b00eb869c23ce019f824985131cc60c570568
Nov 03 13:20:32.022319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.022285    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:32.022392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.022355    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:32.022392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.022383    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:32.107592 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:32.107561    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de14b3f_b00e_4105_b5e9_4b52ef946958.slice/crio-2234b7ec7ed6df1cfd969d098b8b44c2bdec19fcb1eaba0b2e9309675cc65be4 WatchSource:0}: Error finding container 2234b7ec7ed6df1cfd969d098b8b44c2bdec19fcb1eaba0b2e9309675cc65be4: Status 404 returned error can't find the container with id 2234b7ec7ed6df1cfd969d098b8b44c2bdec19fcb1eaba0b2e9309675cc65be4
Nov 03 13:20:32.108600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.108569    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"]
Nov 03 13:20:32.110590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.110554    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:32.110697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.110621    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:32.110697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.110650    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:32.128187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.128119    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" event={"ID":"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1","Type":"ContainerStarted","Data":"8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2"}
Nov 03 13:20:32.128187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.128176    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" event={"ID":"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1","Type":"ContainerStarted","Data":"ae11ed494a894ccab9fb9ee3d92b00eb869c23ce019f824985131cc60c570568"}
Nov 03 13:20:32.129764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.129738    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" event={"ID":"3de14b3f-b00e-4105-b5e9-4b52ef946958","Type":"ContainerStarted","Data":"2234b7ec7ed6df1cfd969d098b8b44c2bdec19fcb1eaba0b2e9309675cc65be4"}
Nov 03 13:20:32.131320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.131244    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9" event={"ID":"bedd069f-adbf-4c74-94d8-ac5782e5e1c2","Type":"ContainerDied","Data":"70a390f489d34d729844202fd2dfeefc66c35ce672430cc8af3bb74ca1bb2d4a"}
Nov 03 13:20:32.131320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.131271    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"
Nov 03 13:20:32.131320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.131294    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a390f489d34d729844202fd2dfeefc66c35ce672430cc8af3bb74ca1bb2d4a"
Nov 03 13:20:32.151707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.151666    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" podStartSLOduration=1.151651277 podStartE2EDuration="1.151651277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:32.149161414 +0000 UTC m=+673.226926440" watchObservedRunningTime="2026-11-03 13:20:32.151651277 +0000 UTC m=+673.229416303"
Nov 03 13:20:32.410197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.410110    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:32.736097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.736062    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:32.938236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:32.925874    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:33.174343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:33.174234    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" event={"ID":"3de14b3f-b00e-4105-b5e9-4b52ef946958","Type":"ContainerStarted","Data":"919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490"}
Nov 03 13:20:33.200038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:33.199980    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" podStartSLOduration=2.199965403 podStartE2EDuration="2.199965403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:33.196731461 +0000 UTC m=+674.274496486" watchObservedRunningTime="2026-11-03 13:20:33.199965403 +0000 UTC m=+674.277730429"
Nov 03 13:20:33.746886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:33.746846    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 13:20:33.747184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:33.747152    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 13:20:33.773105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:33.773067    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:33.841326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:33.841293    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:33.842353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:33.842321    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:33.842583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:33.842557    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 13:20:34.314771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:34.314726    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:34.741583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:34.741550    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 13:20:34.741867 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:34.741838    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 13:20:34.832242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:34.827679    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:34.832535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:34.832499    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 13:20:35.197667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.197588    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:35.307887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.307852    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"]
Nov 03 13:20:35.312030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.312008    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.316890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.316861    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:20:35.317325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.316991    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:20:35.318135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.318112    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:20:35.318279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.318183    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:20:35.319588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.319567    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"]
Nov 03 13:20:35.400726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.400625    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnxp\" (UniqueName: \"kubernetes.io/projected/fd06a2af-8c53-4106-87ec-f18e2e034c5f-kube-api-access-4hnxp\") pod \"maas-api-revoke-sub-f585f393-hwn5w\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.401045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.401022    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f585f393-hwn5w\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.401116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.401067    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f585f393-hwn5w\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.502443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.502409    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnxp\" (UniqueName: \"kubernetes.io/projected/fd06a2af-8c53-4106-87ec-f18e2e034c5f-kube-api-access-4hnxp\") pod \"maas-api-revoke-sub-f585f393-hwn5w\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.502626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.502546    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f585f393-hwn5w\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.502681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.502661    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f585f393-hwn5w\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.503379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.503359    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f585f393-hwn5w\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.504847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.504826    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f585f393-hwn5w\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.511449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.511431    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnxp\" (UniqueName: \"kubernetes.io/projected/fd06a2af-8c53-4106-87ec-f18e2e034c5f-kube-api-access-4hnxp\") pod \"maas-api-revoke-sub-f585f393-hwn5w\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.694237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.693387    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:35.784305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.770762    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 13:20:35.784305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.770831    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 13:20:35.889018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.888358    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:35.889018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:35.888428    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 13:20:36.090330 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:36.090299    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd06a2af_8c53_4106_87ec_f18e2e034c5f.slice/crio-768fbe0f1bc899e16d2654544c7d368972556e8e0f9d1e08de162f240b66fd29 WatchSource:0}: Error finding container 768fbe0f1bc899e16d2654544c7d368972556e8e0f9d1e08de162f240b66fd29: Status 404 returned error can't find the container with id 768fbe0f1bc899e16d2654544c7d368972556e8e0f9d1e08de162f240b66fd29
Nov 03 13:20:36.090923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:36.090902    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"]
Nov 03 13:20:36.194811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:36.194770    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w" event={"ID":"fd06a2af-8c53-4106-87ec-f18e2e034c5f","Type":"ContainerStarted","Data":"768fbe0f1bc899e16d2654544c7d368972556e8e0f9d1e08de162f240b66fd29"}
Nov 03 13:20:36.647387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:36.647300    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:36.740777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:36.740740    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 13:20:36.741088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:36.741046    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 13:20:36.827751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:36.827705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:36.829329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:36.829302    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:36.829573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:36.829533    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 13:20:37.205572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:37.205537    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w" event={"ID":"fd06a2af-8c53-4106-87ec-f18e2e034c5f","Type":"ContainerStarted","Data":"1adc352a42ce16bec0a269c23169363b6744a84082fe1ba7c1ee5e34ee544a69"}
Nov 03 13:20:37.279056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:37.278985    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w" podStartSLOduration=2.278964193 podStartE2EDuration="2.278964193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:37.278564794 +0000 UTC m=+678.356329820" watchObservedRunningTime="2026-11-03 13:20:37.278964193 +0000 UTC m=+678.356729222"
Nov 03 13:20:37.459304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:37.454729    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:37.656530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:37.656477    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:37.744289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:37.744128    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 13:20:37.744289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:37.744200    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 13:20:37.835702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:37.835662    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:37.836024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:37.835983    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 13:20:38.178152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.178066    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"]
Nov 03 13:20:38.182065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.182041    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.189753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.189726    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"]
Nov 03 13:20:38.215345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.215319    5420 generic.go:356] "Generic (PLEG): container finished" podID="fd06a2af-8c53-4106-87ec-f18e2e034c5f" containerID="1adc352a42ce16bec0a269c23169363b6744a84082fe1ba7c1ee5e34ee544a69" exitCode=0
Nov 03 13:20:38.215562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.215477    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w" event={"ID":"fd06a2af-8c53-4106-87ec-f18e2e034c5f","Type":"ContainerDied","Data":"1adc352a42ce16bec0a269c23169363b6744a84082fe1ba7c1ee5e34ee544a69"}
Nov 03 13:20:38.246035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.245995    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:38.298608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.298555    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f0b0fc4-kwzcj\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.298896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.298877    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f0b0fc4-kwzcj\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.299059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.299043    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npsd6\" (UniqueName: \"kubernetes.io/projected/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-kube-api-access-npsd6\") pod \"maas-api-revoke-sub-5f0b0fc4-kwzcj\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.399830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.399799    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f0b0fc4-kwzcj\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.400014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.399851    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f0b0fc4-kwzcj\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.400014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.399906    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsd6\" (UniqueName: \"kubernetes.io/projected/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-kube-api-access-npsd6\") pod \"maas-api-revoke-sub-5f0b0fc4-kwzcj\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.400759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.400728    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f0b0fc4-kwzcj\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.402117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.402094    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f0b0fc4-kwzcj\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.409697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.409669    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsd6\" (UniqueName: \"kubernetes.io/projected/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-kube-api-access-npsd6\") pod \"maas-api-revoke-sub-5f0b0fc4-kwzcj\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.497202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.497172    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:38.736357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.736320    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 13:20:38.736790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.736389    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 13:20:38.782256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.782231    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"]
Nov 03 13:20:38.783823 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:38.783795    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8243f0e7_f7b4_4bd4_9158_317a95cd8d72.slice/crio-4889e5a8a4ef1e705aa66d3353be3dfd0a92d7cbcea111bb7da4f754e5f2d847 WatchSource:0}: Error finding container 4889e5a8a4ef1e705aa66d3353be3dfd0a92d7cbcea111bb7da4f754e5f2d847: Status 404 returned error can't find the container with id 4889e5a8a4ef1e705aa66d3353be3dfd0a92d7cbcea111bb7da4f754e5f2d847
Nov 03 13:20:38.823961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.823929    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 13:20:38.824108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:38.823997    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 13:20:39.105848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.105809    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"]
Nov 03 13:20:39.115235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.112085    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.126279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.124403    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"]
Nov 03 13:20:39.207981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.207942    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29wq\" (UniqueName: \"kubernetes.io/projected/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-kube-api-access-m29wq\") pod \"maas-api-revoke-sub-cead985e-2lq4x\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.208173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.208062    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cead985e-2lq4x\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.208173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.208122    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cead985e-2lq4x\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.238689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.238640    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj" event={"ID":"8243f0e7-f7b4-4bd4-9158-317a95cd8d72","Type":"ContainerStarted","Data":"93eae85c2dec99191d7994ec4469440f9c48d7244c4427c62f3848cdf527b00d"}
Nov 03 13:20:39.238885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.238731    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj" event={"ID":"8243f0e7-f7b4-4bd4-9158-317a95cd8d72","Type":"ContainerStarted","Data":"4889e5a8a4ef1e705aa66d3353be3dfd0a92d7cbcea111bb7da4f754e5f2d847"}
Nov 03 13:20:39.261945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.261877    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj" podStartSLOduration=1.261856885 podStartE2EDuration="1.261856885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:39.256394748 +0000 UTC m=+680.334159777" watchObservedRunningTime="2026-11-03 13:20:39.261856885 +0000 UTC m=+680.339621912"
Nov 03 13:20:39.309188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.309149    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cead985e-2lq4x\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.309336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.309280    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cead985e-2lq4x\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.309422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.309410    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29wq\" (UniqueName: \"kubernetes.io/projected/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-kube-api-access-m29wq\") pod \"maas-api-revoke-sub-cead985e-2lq4x\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.311261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.311233    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cead985e-2lq4x\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.313474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.313443    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cead985e-2lq4x\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.328226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.323784    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29wq\" (UniqueName: \"kubernetes.io/projected/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-kube-api-access-m29wq\") pod \"maas-api-revoke-sub-cead985e-2lq4x\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.401416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.401389    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:39.443190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.443159    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:39.513737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.512070    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd06a2af-8c53-4106-87ec-f18e2e034c5f-kube-api-access-4hnxp\" (UniqueName: \"kubernetes.io/projected/fd06a2af-8c53-4106-87ec-f18e2e034c5f-kube-api-access-4hnxp\") pod \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") "
Nov 03 13:20:39.513737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.512192    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-openshift-ca\") pod \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") "
Nov 03 13:20:39.513737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.512317    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-xks-ca\") pod \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\" (UID: \"fd06a2af-8c53-4106-87ec-f18e2e034c5f\") "
Nov 03 13:20:39.513737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.513608    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-openshift-ca" pod "fd06a2af-8c53-4106-87ec-f18e2e034c5f" (UID: "fd06a2af-8c53-4106-87ec-f18e2e034c5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:39.526090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.526053    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-xks-ca" pod "fd06a2af-8c53-4106-87ec-f18e2e034c5f" (UID: "fd06a2af-8c53-4106-87ec-f18e2e034c5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:39.535947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.535901    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd06a2af-8c53-4106-87ec-f18e2e034c5f-kube-api-access-4hnxp" pod "fd06a2af-8c53-4106-87ec-f18e2e034c5f" (UID: "fd06a2af-8c53-4106-87ec-f18e2e034c5f"). InnerVolumeSpecName "kube-api-access-4hnxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:39.613651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.613566    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:39.613651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.613597    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd06a2af-8c53-4106-87ec-f18e2e034c5f-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:39.613651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.613608    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hnxp\" (UniqueName: \"kubernetes.io/projected/fd06a2af-8c53-4106-87ec-f18e2e034c5f-kube-api-access-4hnxp\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:39.741220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.741179    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"]
Nov 03 13:20:39.742902 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:39.742873    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9441c7_9bab_4cd2_aacc_0223c8d1e14f.slice/crio-8967c13c094493d09d0ffc94eb5dd29b5d94cf8c88efea30edf0965339123d34 WatchSource:0}: Error finding container 8967c13c094493d09d0ffc94eb5dd29b5d94cf8c88efea30edf0965339123d34: Status 404 returned error can't find the container with id 8967c13c094493d09d0ffc94eb5dd29b5d94cf8c88efea30edf0965339123d34
Nov 03 13:20:39.829085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.828309    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:39.829085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.829042    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:39.829982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.829953    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:20:39.851416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.850914    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:39.851755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.851732    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:39.852613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.852589    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:20:39.867654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:39.867594    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:40.246645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:40.246617    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"
Nov 03 13:20:40.246835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:40.246648    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w" event={"ID":"fd06a2af-8c53-4106-87ec-f18e2e034c5f","Type":"ContainerDied","Data":"768fbe0f1bc899e16d2654544c7d368972556e8e0f9d1e08de162f240b66fd29"}
Nov 03 13:20:40.246835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:40.246692    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="768fbe0f1bc899e16d2654544c7d368972556e8e0f9d1e08de162f240b66fd29"
Nov 03 13:20:40.248317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:40.248292    5420 generic.go:356] "Generic (PLEG): container finished" podID="8243f0e7-f7b4-4bd4-9158-317a95cd8d72" containerID="93eae85c2dec99191d7994ec4469440f9c48d7244c4427c62f3848cdf527b00d" exitCode=0
Nov 03 13:20:40.248434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:40.248363    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj" event={"ID":"8243f0e7-f7b4-4bd4-9158-317a95cd8d72","Type":"ContainerDied","Data":"93eae85c2dec99191d7994ec4469440f9c48d7244c4427c62f3848cdf527b00d"}
Nov 03 13:20:40.249847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:40.249824    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x" event={"ID":"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f","Type":"ContainerStarted","Data":"25007adaafed1af6ab03d19977f4dab783d3a6e932076724076bc5c283fee3aa"}
Nov 03 13:20:40.249939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:40.249874    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x" event={"ID":"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f","Type":"ContainerStarted","Data":"8967c13c094493d09d0ffc94eb5dd29b5d94cf8c88efea30edf0965339123d34"}
Nov 03 13:20:40.287587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:40.287510    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x" podStartSLOduration=1.287491957 podStartE2EDuration="1.287491957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:40.282082411 +0000 UTC m=+681.359847438" watchObservedRunningTime="2026-11-03 13:20:40.287491957 +0000 UTC m=+681.365256985"
Nov 03 13:20:41.238300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.238258    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:41.259596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.259566    5420 generic.go:356] "Generic (PLEG): container finished" podID="7b9441c7-9bab-4cd2-aacc-0223c8d1e14f" containerID="25007adaafed1af6ab03d19977f4dab783d3a6e932076724076bc5c283fee3aa" exitCode=0
Nov 03 13:20:41.259765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.259650    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x" event={"ID":"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f","Type":"ContainerDied","Data":"25007adaafed1af6ab03d19977f4dab783d3a6e932076724076bc5c283fee3aa"}
Nov 03 13:20:41.448699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.448671    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:41.533852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.533771    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-kube-api-access-npsd6\" (UniqueName: \"kubernetes.io/projected/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-kube-api-access-npsd6\") pod \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") "
Nov 03 13:20:41.534008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.533922    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-openshift-ca\") pod \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") "
Nov 03 13:20:41.534080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.534063    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-xks-ca\") pod \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\" (UID: \"8243f0e7-f7b4-4bd4-9158-317a95cd8d72\") "
Nov 03 13:20:41.534330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.534300    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-openshift-ca" pod "8243f0e7-f7b4-4bd4-9158-317a95cd8d72" (UID: "8243f0e7-f7b4-4bd4-9158-317a95cd8d72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:41.534584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.534566    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:41.535937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.535910    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-kube-api-access-npsd6" pod "8243f0e7-f7b4-4bd4-9158-317a95cd8d72" (UID: "8243f0e7-f7b4-4bd4-9158-317a95cd8d72"). InnerVolumeSpecName "kube-api-access-npsd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:41.536104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.536080    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-xks-ca" pod "8243f0e7-f7b4-4bd4-9158-317a95cd8d72" (UID: "8243f0e7-f7b4-4bd4-9158-317a95cd8d72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:41.635638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.635605    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npsd6\" (UniqueName: \"kubernetes.io/projected/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-kube-api-access-npsd6\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:41.635638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.635634    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8243f0e7-f7b4-4bd4-9158-317a95cd8d72-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:41.883024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:41.882949    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:42.043105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.043073    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:42.278602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.278563    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj" event={"ID":"8243f0e7-f7b4-4bd4-9158-317a95cd8d72","Type":"ContainerDied","Data":"4889e5a8a4ef1e705aa66d3353be3dfd0a92d7cbcea111bb7da4f754e5f2d847"}
Nov 03 13:20:42.279005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.278641    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4889e5a8a4ef1e705aa66d3353be3dfd0a92d7cbcea111bb7da4f754e5f2d847"
Nov 03 13:20:42.279005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.278574    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"
Nov 03 13:20:42.423769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.423740    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:42.443403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.443376    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-xks-ca\") pod \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") "
Nov 03 13:20:42.443583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.443468    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-openshift-ca\") pod \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") "
Nov 03 13:20:42.443583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.443538    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-kube-api-access-m29wq\" (UniqueName: \"kubernetes.io/projected/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-kube-api-access-m29wq\") pod \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\" (UID: \"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f\") "
Nov 03 13:20:42.443835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.443811    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-openshift-ca" pod "7b9441c7-9bab-4cd2-aacc-0223c8d1e14f" (UID: "7b9441c7-9bab-4cd2-aacc-0223c8d1e14f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:42.443975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.443958    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:42.445802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.445544    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-xks-ca" pod "7b9441c7-9bab-4cd2-aacc-0223c8d1e14f" (UID: "7b9441c7-9bab-4cd2-aacc-0223c8d1e14f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:42.445996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.445961    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-kube-api-access-m29wq" pod "7b9441c7-9bab-4cd2-aacc-0223c8d1e14f" (UID: "7b9441c7-9bab-4cd2-aacc-0223c8d1e14f"). InnerVolumeSpecName "kube-api-access-m29wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:42.544947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.544852    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m29wq\" (UniqueName: \"kubernetes.io/projected/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-kube-api-access-m29wq\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:42.544947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.544889    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:42.896326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:42.896242    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:43.285978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.285946    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x" event={"ID":"7b9441c7-9bab-4cd2-aacc-0223c8d1e14f","Type":"ContainerDied","Data":"8967c13c094493d09d0ffc94eb5dd29b5d94cf8c88efea30edf0965339123d34"}
Nov 03 13:20:43.285978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.285970    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"
Nov 03 13:20:43.286492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.285995    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8967c13c094493d09d0ffc94eb5dd29b5d94cf8c88efea30edf0965339123d34"
Nov 03 13:20:43.562261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562179    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"]
Nov 03 13:20:43.562754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562737    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8243f0e7-f7b4-4bd4-9158-317a95cd8d72" containerName="revoke-keys"
Nov 03 13:20:43.562801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562756    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8243f0e7-f7b4-4bd4-9158-317a95cd8d72" containerName="revoke-keys"
Nov 03 13:20:43.562801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562773    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7b9441c7-9bab-4cd2-aacc-0223c8d1e14f" containerName="revoke-keys"
Nov 03 13:20:43.562801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562778    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9441c7-9bab-4cd2-aacc-0223c8d1e14f" containerName="revoke-keys"
Nov 03 13:20:43.562894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562811    5420 cpu_manager.go:414] "containerMap: removing container" podUID="fd06a2af-8c53-4106-87ec-f18e2e034c5f" containerName="revoke-keys"
Nov 03 13:20:43.562894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562817    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd06a2af-8c53-4106-87ec-f18e2e034c5f" containerName="revoke-keys"
Nov 03 13:20:43.562978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562902    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9441c7-9bab-4cd2-aacc-0223c8d1e14f" containerName="revoke-keys"
Nov 03 13:20:43.562978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562914    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd06a2af-8c53-4106-87ec-f18e2e034c5f" containerName="revoke-keys"
Nov 03 13:20:43.562978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.562923    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8243f0e7-f7b4-4bd4-9158-317a95cd8d72" containerName="revoke-keys"
Nov 03 13:20:43.566784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.566762    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:43.572740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.572699    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ded926\""
Nov 03 13:20:43.574626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.574590    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:43.577078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.577040    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"]
Nov 03 13:20:43.638573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.638542    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"]
Nov 03 13:20:43.642359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.642342    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:43.650532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.650505    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ded926-dockercfg-fzn88\""
Nov 03 13:20:43.652795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.652771    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"]
Nov 03 13:20:43.658358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.658334    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290a73e4-7a48-4e53-bb52-d49caa66f19f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2\" (UID: \"290a73e4-7a48-4e53-bb52-d49caa66f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:43.658441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.658369    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76lz4\" (UniqueName: \"kubernetes.io/projected/290a73e4-7a48-4e53-bb52-d49caa66f19f-kube-api-access-76lz4\") pod \"payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2\" (UID: \"290a73e4-7a48-4e53-bb52-d49caa66f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:43.732656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.732611    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:43.759859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.759816    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290a73e4-7a48-4e53-bb52-d49caa66f19f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2\" (UID: \"290a73e4-7a48-4e53-bb52-d49caa66f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:43.760085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.759890    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76lz4\" (UniqueName: \"kubernetes.io/projected/290a73e4-7a48-4e53-bb52-d49caa66f19f-kube-api-access-76lz4\") pod \"payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2\" (UID: \"290a73e4-7a48-4e53-bb52-d49caa66f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:43.760171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.760148    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c61d8e-873c-4198-a6d3-b75706be6bf3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7\" (UID: \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:43.760247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.760224    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj46x\" (UniqueName: \"kubernetes.io/projected/a1c61d8e-873c-4198-a6d3-b75706be6bf3-kube-api-access-kj46x\") pod \"payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7\" (UID: \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:43.761573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.761498    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290a73e4-7a48-4e53-bb52-d49caa66f19f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2\" (UID: \"290a73e4-7a48-4e53-bb52-d49caa66f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:43.770452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.770421    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76lz4\" (UniqueName: \"kubernetes.io/projected/290a73e4-7a48-4e53-bb52-d49caa66f19f-kube-api-access-76lz4\") pod \"payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2\" (UID: \"290a73e4-7a48-4e53-bb52-d49caa66f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:43.862001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.861912    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c61d8e-873c-4198-a6d3-b75706be6bf3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7\" (UID: \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:43.862001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.861969    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj46x\" (UniqueName: \"kubernetes.io/projected/a1c61d8e-873c-4198-a6d3-b75706be6bf3-kube-api-access-kj46x\") pod \"payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7\" (UID: \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:43.862604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.862582    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c61d8e-873c-4198-a6d3-b75706be6bf3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7\" (UID: \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:43.870719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.870690    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj46x\" (UniqueName: \"kubernetes.io/projected/a1c61d8e-873c-4198-a6d3-b75706be6bf3-kube-api-access-kj46x\") pod \"payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7\" (UID: \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:43.885133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.885111    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:43.956464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:43.956430    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:44.161263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:44.161239    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"]
Nov 03 13:20:44.162107 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:44.162083    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290a73e4_7a48_4e53_bb52_d49caa66f19f.slice/crio-5a30e3babcb62d2389f42c476f732e87b3085cb7a1e6a1247978dbe695545daa WatchSource:0}: Error finding container 5a30e3babcb62d2389f42c476f732e87b3085cb7a1e6a1247978dbe695545daa: Status 404 returned error can't find the container with id 5a30e3babcb62d2389f42c476f732e87b3085cb7a1e6a1247978dbe695545daa
Nov 03 13:20:44.245810 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:44.245786    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"]
Nov 03 13:20:44.246649 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:44.246628    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c61d8e_873c_4198_a6d3_b75706be6bf3.slice/crio-31f111aae97b3a750e7f8f08df7e9b7dfd91ce8708727147f7eb4e2d2f8011d5 WatchSource:0}: Error finding container 31f111aae97b3a750e7f8f08df7e9b7dfd91ce8708727147f7eb4e2d2f8011d5: Status 404 returned error can't find the container with id 31f111aae97b3a750e7f8f08df7e9b7dfd91ce8708727147f7eb4e2d2f8011d5
Nov 03 13:20:44.294358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:44.294328    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2" event={"ID":"290a73e4-7a48-4e53-bb52-d49caa66f19f","Type":"ContainerStarted","Data":"8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d"}
Nov 03 13:20:44.294772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:44.294385    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2" event={"ID":"290a73e4-7a48-4e53-bb52-d49caa66f19f","Type":"ContainerStarted","Data":"5a30e3babcb62d2389f42c476f732e87b3085cb7a1e6a1247978dbe695545daa"}
Nov 03 13:20:44.294772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:44.294416    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:44.295890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:44.295851    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7" event={"ID":"a1c61d8e-873c-4198-a6d3-b75706be6bf3","Type":"ContainerStarted","Data":"31f111aae97b3a750e7f8f08df7e9b7dfd91ce8708727147f7eb4e2d2f8011d5"}
Nov 03 13:20:44.310877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:44.310834    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2" podStartSLOduration=1.3108215300000001 podStartE2EDuration="1.31082153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:44.310187882 +0000 UTC m=+685.387952907" watchObservedRunningTime="2026-11-03 13:20:44.31082153 +0000 UTC m=+685.388586593"
Nov 03 13:20:44.791477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:44.791417    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:45.191156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:45.191067    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:20:45.303731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:45.303696    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7" event={"ID":"a1c61d8e-873c-4198-a6d3-b75706be6bf3","Type":"ContainerStarted","Data":"12b8fcb0381da3c8fee87f3df375a227e884d001b413395138b2b22ed709e078"}
Nov 03 13:20:45.304176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:45.303765    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:45.322471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:45.322415    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7" podStartSLOduration=2.32239864 podStartE2EDuration="2.32239864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:45.321330561 +0000 UTC m=+686.399095587" watchObservedRunningTime="2026-11-03 13:20:45.32239864 +0000 UTC m=+686.400163666"
Nov 03 13:20:46.355240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.342144    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"]
Nov 03 13:20:46.364234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.359887    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.397236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.390369    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"]
Nov 03 13:20:46.401234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.400508    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.401234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.400554    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.401234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.400587    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.401234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.400735    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.401234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.400757    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.401234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.400790    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.401234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.400847    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.401234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.400921    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.401234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.400951    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsgpp\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-kube-api-access-nsgpp\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.519252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.517895    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.519252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.517950    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.519252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.517986    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.519252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.518097    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.519252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.518122    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.519252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.518156    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.519252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.518227    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.519252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.518296    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.519252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.518330    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsgpp\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-kube-api-access-nsgpp\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.557261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.522143    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.557261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.529574    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.557261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.530323    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.557261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.530584    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.557261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.530825    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.575035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.575005    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"]
Nov 03 13:20:46.713239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.709816    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:46.768259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.753225    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsgpp\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-kube-api-access-nsgpp\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.782272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.768557    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.794227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.790575    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.794227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.791090    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:46.794227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.791524    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.807391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.795176    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"]
Nov 03 13:20:46.833303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.831882    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.833303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.831974    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.833303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.832014    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.833303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.832040    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j94lg\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-kube-api-access-j94lg\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.833303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.832104    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.833303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.832156    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.833303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.832236    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8069b68c-2d60-473b-968c-9013630f3a00-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.833303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.832267    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8069b68c-2d60-473b-968c-9013630f3a00-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.833303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.832341    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.941244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.939298    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.941244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.939784    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.951234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.948139    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8069b68c-2d60-473b-968c-9013630f3a00-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.951234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.947377    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8069b68c-2d60-473b-968c-9013630f3a00-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.955233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.952291    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8069b68c-2d60-473b-968c-9013630f3a00-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.952915    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.970888    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8069b68c-2d60-473b-968c-9013630f3a00-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.971583    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.971958    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.972060    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.972413    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.972346    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.972490    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j94lg\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-kube-api-access-j94lg\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.973166    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.973882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.973498    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.981585 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.981484    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.989998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.989816    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:46.991999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:46.991972    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j94lg\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-kube-api-access-j94lg\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:47.029007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.028655    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:47.143814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.143776    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:47.389091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.389061    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"]
Nov 03 13:20:47.391750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.391713    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:47.391949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.391925    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:47.392058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.391979    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:47.485297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.485002    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"]
Nov 03 13:20:47.489059 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:47.489030    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8069b68c_2d60_473b_968c_9013630f3a00.slice/crio-68b8249689450c731c21310b08cab5d2f18e8228908dbc4513c1ca285218ddbf WatchSource:0}: Error finding container 68b8249689450c731c21310b08cab5d2f18e8228908dbc4513c1ca285218ddbf: Status 404 returned error can't find the container with id 68b8249689450c731c21310b08cab5d2f18e8228908dbc4513c1ca285218ddbf
Nov 03 13:20:47.491575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.491538    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:47.491685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.491618    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:47.491685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:47.491663    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:20:48.287343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:48.287306    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:48.395782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:48.395074    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" event={"ID":"adb963ee-4cb9-4f6a-945d-5fbbf05af5da","Type":"ContainerStarted","Data":"f301f0861de0d3848f92f8e05c6e465b513e09438c9a678dbac69ce8f36be18d"}
Nov 03 13:20:48.395782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:48.395126    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" event={"ID":"adb963ee-4cb9-4f6a-945d-5fbbf05af5da","Type":"ContainerStarted","Data":"d9c57d3c0e09d5441753f7c6602d409dba9de52cc70f42454b7c07ffabf3c635"}
Nov 03 13:20:48.398460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:48.398225    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" event={"ID":"8069b68c-2d60-473b-968c-9013630f3a00","Type":"ContainerStarted","Data":"af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0"}
Nov 03 13:20:48.398460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:48.398274    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" event={"ID":"8069b68c-2d60-473b-968c-9013630f3a00","Type":"ContainerStarted","Data":"68b8249689450c731c21310b08cab5d2f18e8228908dbc4513c1ca285218ddbf"}
Nov 03 13:20:48.473222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:48.473136    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" podStartSLOduration=2.473116412 podStartE2EDuration="2.473116412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:48.428752899 +0000 UTC m=+689.506517925" watchObservedRunningTime="2026-11-03 13:20:48.473116412 +0000 UTC m=+689.550881439"
Nov 03 13:20:48.475417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:48.475260    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" podStartSLOduration=2.475244814 podStartE2EDuration="2.475244814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:48.472709738 +0000 UTC m=+689.550474768" watchObservedRunningTime="2026-11-03 13:20:48.475244814 +0000 UTC m=+689.553009842"
Nov 03 13:20:49.031874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.031838    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:49.032259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.032233    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:20:49.032470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.032442    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 13:20:49.156277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.156247    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:49.167367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.167334    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:20:49.167635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.167605    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 13:20:49.482985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.482948    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"]
Nov 03 13:20:49.487268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.487241    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.488138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.488074    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"]
Nov 03 13:20:49.492537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.492511    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:20:49.493299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.492612    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:20:49.493299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.492658    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:20:49.493299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.492676    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:20:49.617859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.617829    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4074262-gstsl\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.618037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.617882    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4074262-gstsl\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.618082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.618056    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s87t\" (UniqueName: \"kubernetes.io/projected/c44ac396-14d6-42ff-8388-b440acd7aa46-kube-api-access-7s87t\") pod \"maas-api-revoke-sub-a4074262-gstsl\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.719708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.719670    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4074262-gstsl\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.719861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.719728    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4074262-gstsl\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.719861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.719817    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s87t\" (UniqueName: \"kubernetes.io/projected/c44ac396-14d6-42ff-8388-b440acd7aa46-kube-api-access-7s87t\") pod \"maas-api-revoke-sub-a4074262-gstsl\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.746288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.734055    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4074262-gstsl\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.769235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.750303    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4074262-gstsl\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.772348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.772316    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s87t\" (UniqueName: \"kubernetes.io/projected/c44ac396-14d6-42ff-8388-b440acd7aa46-kube-api-access-7s87t\") pod \"maas-api-revoke-sub-a4074262-gstsl\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:49.780241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.780184    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:49.805252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:49.805227    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:50.037307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.033640    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:20:50.037307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.033714    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 13:20:50.086232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.086183    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"]
Nov 03 13:20:50.091314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.091292    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.098183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.098162    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"]
Nov 03 13:20:50.145617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.145574    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:20:50.145823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.145639    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 13:20:50.208450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.208409    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"]
Nov 03 13:20:50.208580 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:50.208562    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44ac396_14d6_42ff_8388_b440acd7aa46.slice/crio-8bf2989ec4a63e231a2242eaa3afd8fac4fd012fe6e8f7fabe213a961b691dd6 WatchSource:0}: Error finding container 8bf2989ec4a63e231a2242eaa3afd8fac4fd012fe6e8f7fabe213a961b691dd6: Status 404 returned error can't find the container with id 8bf2989ec4a63e231a2242eaa3afd8fac4fd012fe6e8f7fabe213a961b691dd6
Nov 03 13:20:50.270263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.269963    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtxtx\" (UniqueName: \"kubernetes.io/projected/59bb4563-2d22-452e-ae90-68c8c477ce7a-kube-api-access-wtxtx\") pod \"maas-api-revoke-sub-ec84dd02-8mntd\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.270263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.270119    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec84dd02-8mntd\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.270263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.270206    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec84dd02-8mntd\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.382287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.374883    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec84dd02-8mntd\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.382287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.375033    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtxtx\" (UniqueName: \"kubernetes.io/projected/59bb4563-2d22-452e-ae90-68c8c477ce7a-kube-api-access-wtxtx\") pod \"maas-api-revoke-sub-ec84dd02-8mntd\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.382287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.375187    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec84dd02-8mntd\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.382287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.376627    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec84dd02-8mntd\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.412808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.412747    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl" event={"ID":"c44ac396-14d6-42ff-8388-b440acd7aa46","Type":"ContainerStarted","Data":"8bf2989ec4a63e231a2242eaa3afd8fac4fd012fe6e8f7fabe213a961b691dd6"}
Nov 03 13:20:50.438237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.434322    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec84dd02-8mntd\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.450237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.449259    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtxtx\" (UniqueName: \"kubernetes.io/projected/59bb4563-2d22-452e-ae90-68c8c477ce7a-kube-api-access-wtxtx\") pod \"maas-api-revoke-sub-ec84dd02-8mntd\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:50.707020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:50.706989    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:51.034246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.033331    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:20:51.034246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.033392    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 13:20:51.100270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.100246    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"]
Nov 03 13:20:51.100419 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:51.100395    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bb4563_2d22_452e_ae90_68c8c477ce7a.slice/crio-1da5b0677941cd26b661197c406cb8f057fc53ce9f5617e82d97dd6c670e7221 WatchSource:0}: Error finding container 1da5b0677941cd26b661197c406cb8f057fc53ce9f5617e82d97dd6c670e7221: Status 404 returned error can't find the container with id 1da5b0677941cd26b661197c406cb8f057fc53ce9f5617e82d97dd6c670e7221
Nov 03 13:20:51.147238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.146134    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:20:51.147238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.146195    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 13:20:51.256234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.246041    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:51.382899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.382785    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:51.496240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.495106    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd" event={"ID":"59bb4563-2d22-452e-ae90-68c8c477ce7a","Type":"ContainerStarted","Data":"1da5b0677941cd26b661197c406cb8f057fc53ce9f5617e82d97dd6c670e7221"}
Nov 03 13:20:51.497268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.497224    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl" event={"ID":"c44ac396-14d6-42ff-8388-b440acd7aa46","Type":"ContainerStarted","Data":"21181cd12e1744d7ae071240bb4831a661e1fccce4e966f70a0722f22a08256a"}
Nov 03 13:20:51.510822 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:20:51.510793    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44ac396_14d6_42ff_8388_b440acd7aa46.slice/crio-conmon-21181cd12e1744d7ae071240bb4831a661e1fccce4e966f70a0722f22a08256a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:20:51.517616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.517570    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl" podStartSLOduration=2.517557412 podStartE2EDuration="2.517557412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:51.514310863 +0000 UTC m=+692.592075892" watchObservedRunningTime="2026-11-03 13:20:51.517557412 +0000 UTC m=+692.595322438"
Nov 03 13:20:51.532244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:51.532060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:52.029667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:52.029628    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:20:52.030291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:52.030256    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 13:20:52.157303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:52.147769    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:20:52.157303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:52.147842    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 13:20:52.505813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:52.505779    5420 generic.go:356] "Generic (PLEG): container finished" podID="59bb4563-2d22-452e-ae90-68c8c477ce7a" containerID="dbe2513bf4cdf9672f9a69fba158c170a00cbb0d421f6546bd2a06ef235bbd05" exitCode=0
Nov 03 13:20:52.505963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:52.505862    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd" event={"ID":"59bb4563-2d22-452e-ae90-68c8c477ce7a","Type":"ContainerDied","Data":"dbe2513bf4cdf9672f9a69fba158c170a00cbb0d421f6546bd2a06ef235bbd05"}
Nov 03 13:20:52.507915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:52.507896    5420 generic.go:356] "Generic (PLEG): container finished" podID="c44ac396-14d6-42ff-8388-b440acd7aa46" containerID="21181cd12e1744d7ae071240bb4831a661e1fccce4e966f70a0722f22a08256a" exitCode=0
Nov 03 13:20:52.508023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:52.508011    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl" event={"ID":"c44ac396-14d6-42ff-8388-b440acd7aa46","Type":"ContainerDied","Data":"21181cd12e1744d7ae071240bb4831a661e1fccce4e966f70a0722f22a08256a"}
Nov 03 13:20:52.617672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:52.617572    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:53.029653 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.029616    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:20:53.029917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.029887    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 13:20:53.148694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.148655    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:20:53.149007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.148971    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 13:20:53.506599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.506568    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:53.721574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.721542    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:53.779581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.779555    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:53.834520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.834487    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-xks-ca\") pod \"c44ac396-14d6-42ff-8388-b440acd7aa46\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") "
Nov 03 13:20:53.834705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.834642    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-openshift-ca\") pod \"59bb4563-2d22-452e-ae90-68c8c477ce7a\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") "
Nov 03 13:20:53.834801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.834709    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-openshift-ca\") pod \"c44ac396-14d6-42ff-8388-b440acd7aa46\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") "
Nov 03 13:20:53.834801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.834728    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bb4563-2d22-452e-ae90-68c8c477ce7a-kube-api-access-wtxtx\" (UniqueName: \"kubernetes.io/projected/59bb4563-2d22-452e-ae90-68c8c477ce7a-kube-api-access-wtxtx\") pod \"59bb4563-2d22-452e-ae90-68c8c477ce7a\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") "
Nov 03 13:20:53.834801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.834756    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-xks-ca\") pod \"59bb4563-2d22-452e-ae90-68c8c477ce7a\" (UID: \"59bb4563-2d22-452e-ae90-68c8c477ce7a\") "
Nov 03 13:20:53.834963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.834802    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44ac396-14d6-42ff-8388-b440acd7aa46-kube-api-access-7s87t\" (UniqueName: \"kubernetes.io/projected/c44ac396-14d6-42ff-8388-b440acd7aa46-kube-api-access-7s87t\") pod \"c44ac396-14d6-42ff-8388-b440acd7aa46\" (UID: \"c44ac396-14d6-42ff-8388-b440acd7aa46\") "
Nov 03 13:20:53.835249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.835180    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-openshift-ca" pod "c44ac396-14d6-42ff-8388-b440acd7aa46" (UID: "c44ac396-14d6-42ff-8388-b440acd7aa46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:53.835630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.835598    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-openshift-ca" pod "59bb4563-2d22-452e-ae90-68c8c477ce7a" (UID: "59bb4563-2d22-452e-ae90-68c8c477ce7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:53.837561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.837534    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-xks-ca" pod "59bb4563-2d22-452e-ae90-68c8c477ce7a" (UID: "59bb4563-2d22-452e-ae90-68c8c477ce7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:53.838250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.838098    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-xks-ca" pod "c44ac396-14d6-42ff-8388-b440acd7aa46" (UID: "c44ac396-14d6-42ff-8388-b440acd7aa46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:53.838250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.838223    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bb4563-2d22-452e-ae90-68c8c477ce7a-kube-api-access-wtxtx" pod "59bb4563-2d22-452e-ae90-68c8c477ce7a" (UID: "59bb4563-2d22-452e-ae90-68c8c477ce7a"). InnerVolumeSpecName "kube-api-access-wtxtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:53.839701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.839676    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44ac396-14d6-42ff-8388-b440acd7aa46-kube-api-access-7s87t" pod "c44ac396-14d6-42ff-8388-b440acd7aa46" (UID: "c44ac396-14d6-42ff-8388-b440acd7aa46"). InnerVolumeSpecName "kube-api-access-7s87t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:53.868504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.868471    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"]
Nov 03 13:20:53.869368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.869342    5420 cpu_manager.go:414] "containerMap: removing container" podUID="59bb4563-2d22-452e-ae90-68c8c477ce7a" containerName="revoke-keys"
Nov 03 13:20:53.869368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.869367    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bb4563-2d22-452e-ae90-68c8c477ce7a" containerName="revoke-keys"
Nov 03 13:20:53.869572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.869428    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c44ac396-14d6-42ff-8388-b440acd7aa46" containerName="revoke-keys"
Nov 03 13:20:53.869572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.869438    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44ac396-14d6-42ff-8388-b440acd7aa46" containerName="revoke-keys"
Nov 03 13:20:53.869572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.869559    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bb4563-2d22-452e-ae90-68c8c477ce7a" containerName="revoke-keys"
Nov 03 13:20:53.869738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.869583    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44ac396-14d6-42ff-8388-b440acd7aa46" containerName="revoke-keys"
Nov 03 13:20:53.873839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.873816    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:53.883328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.883292    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"]
Nov 03 13:20:53.937029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.936987    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b9a106a-8rfqf\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:53.937230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.937065    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w66qk\" (UniqueName: \"kubernetes.io/projected/4365d6b0-ca51-4863-9ad8-d855850fcead-kube-api-access-w66qk\") pod \"maas-api-revoke-sub-5b9a106a-8rfqf\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:53.937283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.937250    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b9a106a-8rfqf\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:53.937463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.937441    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:53.937536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.937467    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtxtx\" (UniqueName: \"kubernetes.io/projected/59bb4563-2d22-452e-ae90-68c8c477ce7a-kube-api-access-wtxtx\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:53.937536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.937483    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:53.937536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.937499    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s87t\" (UniqueName: \"kubernetes.io/projected/c44ac396-14d6-42ff-8388-b440acd7aa46-kube-api-access-7s87t\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:53.937536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.937513    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44ac396-14d6-42ff-8388-b440acd7aa46-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:53.937536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:53.937528    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59bb4563-2d22-452e-ae90-68c8c477ce7a-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:54.030133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.030036    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 13:20:54.030133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.030105    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 13:20:54.038739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.038701    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b9a106a-8rfqf\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:54.038920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.038773    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w66qk\" (UniqueName: \"kubernetes.io/projected/4365d6b0-ca51-4863-9ad8-d855850fcead-kube-api-access-w66qk\") pod \"maas-api-revoke-sub-5b9a106a-8rfqf\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:54.038981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.038916    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b9a106a-8rfqf\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:54.039948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.039921    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b9a106a-8rfqf\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:54.044459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.044432    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b9a106a-8rfqf\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:54.049290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.049262    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w66qk\" (UniqueName: \"kubernetes.io/projected/4365d6b0-ca51-4863-9ad8-d855850fcead-kube-api-access-w66qk\") pod \"maas-api-revoke-sub-5b9a106a-8rfqf\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:54.145554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.145520    5420 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 13:20:54.145725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.145586    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 13:20:54.201829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.201794    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:54.229269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.225750    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:54.526570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.526534    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"]
Nov 03 13:20:54.527497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.527460    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd" event={"ID":"59bb4563-2d22-452e-ae90-68c8c477ce7a","Type":"ContainerDied","Data":"1da5b0677941cd26b661197c406cb8f057fc53ce9f5617e82d97dd6c670e7221"}
Nov 03 13:20:54.527637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.527521    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1da5b0677941cd26b661197c406cb8f057fc53ce9f5617e82d97dd6c670e7221"
Nov 03 13:20:54.527637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.527596    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"
Nov 03 13:20:54.528328 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:20:54.528297    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4365d6b0_ca51_4863_9ad8_d855850fcead.slice/crio-8cb3f8d1ce839e88129dfd357d4c2bd9fd6a6e567602ed2d771c418ea2f22a77 WatchSource:0}: Error finding container 8cb3f8d1ce839e88129dfd357d4c2bd9fd6a6e567602ed2d771c418ea2f22a77: Status 404 returned error can't find the container with id 8cb3f8d1ce839e88129dfd357d4c2bd9fd6a6e567602ed2d771c418ea2f22a77
Nov 03 13:20:54.531532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.531491    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl" event={"ID":"c44ac396-14d6-42ff-8388-b440acd7aa46","Type":"ContainerDied","Data":"8bf2989ec4a63e231a2242eaa3afd8fac4fd012fe6e8f7fabe213a961b691dd6"}
Nov 03 13:20:54.531628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.531580    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bf2989ec4a63e231a2242eaa3afd8fac4fd012fe6e8f7fabe213a961b691dd6"
Nov 03 13:20:54.531705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:54.531690    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"
Nov 03 13:20:54.733775 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:20:54.733745    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:20:55.034099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.034072    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:55.034507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.034428    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:55.035269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.035254    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:20:55.149333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.149260    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:55.149744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.149723    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:55.150755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.150731    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:20:55.202195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.202154    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:55.350236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.349889    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:20:55.543782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.543746    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf" event={"ID":"4365d6b0-ca51-4863-9ad8-d855850fcead","Type":"ContainerStarted","Data":"b393ef507abc25cc52123b541e2e60c741b2b0dd419ddc950efd3daecdf284d5"}
Nov 03 13:20:55.544054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:55.544032    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf" event={"ID":"4365d6b0-ca51-4863-9ad8-d855850fcead","Type":"ContainerStarted","Data":"8cb3f8d1ce839e88129dfd357d4c2bd9fd6a6e567602ed2d771c418ea2f22a77"}
Nov 03 13:20:56.344285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:56.344253    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:56.363592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:56.362440    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:20:56.442240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:56.440083    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf" podStartSLOduration=3.440058507 podStartE2EDuration="3.440058507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:55.563591318 +0000 UTC m=+696.641356345" watchObservedRunningTime="2026-11-03 13:20:56.440058507 +0000 UTC m=+697.517823533"
Nov 03 13:20:56.653289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:56.632511    5420 generic.go:356] "Generic (PLEG): container finished" podID="4365d6b0-ca51-4863-9ad8-d855850fcead" containerID="b393ef507abc25cc52123b541e2e60c741b2b0dd419ddc950efd3daecdf284d5" exitCode=0
Nov 03 13:20:56.653289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:56.633766    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf" event={"ID":"4365d6b0-ca51-4863-9ad8-d855850fcead","Type":"ContainerDied","Data":"b393ef507abc25cc52123b541e2e60c741b2b0dd419ddc950efd3daecdf284d5"}
Nov 03 13:20:57.115812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.115776    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:57.786414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.786389    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:57.803798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.803766    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:57.815417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.815327    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-xks-ca\") pod \"4365d6b0-ca51-4863-9ad8-d855850fcead\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") "
Nov 03 13:20:57.815625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.815606    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-openshift-ca\") pod \"4365d6b0-ca51-4863-9ad8-d855850fcead\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") "
Nov 03 13:20:57.815779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.815764    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4365d6b0-ca51-4863-9ad8-d855850fcead-kube-api-access-w66qk\" (UniqueName: \"kubernetes.io/projected/4365d6b0-ca51-4863-9ad8-d855850fcead-kube-api-access-w66qk\") pod \"4365d6b0-ca51-4863-9ad8-d855850fcead\" (UID: \"4365d6b0-ca51-4863-9ad8-d855850fcead\") "
Nov 03 13:20:57.817863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.816085    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-openshift-ca" pod "4365d6b0-ca51-4863-9ad8-d855850fcead" (UID: "4365d6b0-ca51-4863-9ad8-d855850fcead"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:20:57.817863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.816320    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:57.818251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.818196    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-xks-ca" pod "4365d6b0-ca51-4863-9ad8-d855850fcead" (UID: "4365d6b0-ca51-4863-9ad8-d855850fcead"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:57.819547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.819513    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4365d6b0-ca51-4863-9ad8-d855850fcead-kube-api-access-w66qk" pod "4365d6b0-ca51-4863-9ad8-d855850fcead" (UID: "4365d6b0-ca51-4863-9ad8-d855850fcead"). InnerVolumeSpecName "kube-api-access-w66qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:57.917403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.917317    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w66qk\" (UniqueName: \"kubernetes.io/projected/4365d6b0-ca51-4863-9ad8-d855850fcead-kube-api-access-w66qk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:57.917403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:57.917351    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365d6b0-ca51-4863-9ad8-d855850fcead-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:58.395790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:58.395751    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:58.647741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:58.647655    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf" event={"ID":"4365d6b0-ca51-4863-9ad8-d855850fcead","Type":"ContainerDied","Data":"8cb3f8d1ce839e88129dfd357d4c2bd9fd6a6e567602ed2d771c418ea2f22a77"}
Nov 03 13:20:58.647741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:58.647697    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"
Nov 03 13:20:58.647741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:58.647708    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cb3f8d1ce839e88129dfd357d4c2bd9fd6a6e567602ed2d771c418ea2f22a77"
Nov 03 13:20:59.104145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:59.104110    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:59.849920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:20:59.849883    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:00.519525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:00.519497    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:01.231881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.231842    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:01.383705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.383673    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"]
Nov 03 13:21:01.384202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.384188    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4365d6b0-ca51-4863-9ad8-d855850fcead" containerName="revoke-keys"
Nov 03 13:21:01.384294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.384203    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4365d6b0-ca51-4863-9ad8-d855850fcead" containerName="revoke-keys"
Nov 03 13:21:01.384331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.384313    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4365d6b0-ca51-4863-9ad8-d855850fcead" containerName="revoke-keys"
Nov 03 13:21:01.387710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.387694    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.407716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.407687    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"]
Nov 03 13:21:01.435725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.435691    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"]
Nov 03 13:21:01.440724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.440695    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.456186    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"]
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.460784    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.460835    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.460892    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.460925    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.461022    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.461055    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndhkp\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-kube-api-access-ndhkp\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.461073    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.461090    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.462457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.461122    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd652cb-d578-4f95-b538-ed75795b5ddd-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.561832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.561752    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.561832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.561787    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.562326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.561888    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.562326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.561962    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.562326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.561999    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndhkp\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-kube-api-access-ndhkp\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.562326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562035    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.562326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562063    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.562326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562111    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd652cb-d578-4f95-b538-ed75795b5ddd-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.562326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562167    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4s7r\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-kube-api-access-d4s7r\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.562326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562228    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.562326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562265    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.562767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562348    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.562767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562477    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb216de6-36b1-4280-b763-fb951b3bb9d6-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.562767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562529    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.562767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562599    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.562767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562674    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.562767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562708    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.563075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562786    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.563075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.562787    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.563075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.563055    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.563228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.563143    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.563228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.563169    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd652cb-d578-4f95-b538-ed75795b5ddd-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.563677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.563650    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.564858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.564828    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.565106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.565085    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.570983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.570951    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndhkp\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-kube-api-access-ndhkp\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.571076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.571062    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.664058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664019    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664137    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4s7r\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-kube-api-access-d4s7r\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664175    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664206    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664317    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664351    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb216de6-36b1-4280-b763-fb951b3bb9d6-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664444    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664476    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664529    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664554    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664664    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664678    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.664996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.664974    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.665062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.665047    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb216de6-36b1-4280-b763-fb951b3bb9d6-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.666878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.666853    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.667112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.667096    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.673285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.673263    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.673376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.673347    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4s7r\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-kube-api-access-d4s7r\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.701293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.701254    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:01.771255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.768125    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:01.992436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.992412    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"]
Nov 03 13:21:01.995065 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:01.995037    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd652cb_d578_4f95_b538_ed75795b5ddd.slice/crio-1dcb92e2f68e2fa4a873bcbd624a0082ddeb75a47edc74aeeb79eb00b8e2a164 WatchSource:0}: Error finding container 1dcb92e2f68e2fa4a873bcbd624a0082ddeb75a47edc74aeeb79eb00b8e2a164: Status 404 returned error can't find the container with id 1dcb92e2f68e2fa4a873bcbd624a0082ddeb75a47edc74aeeb79eb00b8e2a164
Nov 03 13:21:01.998195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.998162    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:01.998281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.998264    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:01.998334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:01.998300    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:02.069894 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:02.069861    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb216de6_36b1_4280_b763_fb951b3bb9d6.slice/crio-a752f1f69b5bec658596900f506b9d0e9cf3109806b392a7e31ab8ca1c5f37f3 WatchSource:0}: Error finding container a752f1f69b5bec658596900f506b9d0e9cf3109806b392a7e31ab8ca1c5f37f3: Status 404 returned error can't find the container with id a752f1f69b5bec658596900f506b9d0e9cf3109806b392a7e31ab8ca1c5f37f3
Nov 03 13:21:02.070036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.070009    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"]
Nov 03 13:21:02.072553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.072513    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:02.072650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.072604    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:02.072690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.072650    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:02.094226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.094185    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:02.162335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.162296    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"]
Nov 03 13:21:02.177308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.177275    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"]
Nov 03 13:21:02.177496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.177395    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.195168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.188524    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:21:02.195168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.188669    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:21:02.195168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.188711    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:21:02.195168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.188819    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:21:02.266243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.266199    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"]
Nov 03 13:21:02.270465 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.270444    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.282124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.281131    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b8ead43-jf5d8\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.282124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.281274    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjv2\" (UniqueName: \"kubernetes.io/projected/ff68e187-075b-46fb-8977-025f0f0123b7-kube-api-access-4bjv2\") pod \"maas-api-revoke-sub-8b8ead43-jf5d8\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.282124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.281306    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b8ead43-jf5d8\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.286972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.286938    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"]
Nov 03 13:21:02.382647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.382546    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjv2\" (UniqueName: \"kubernetes.io/projected/ff68e187-075b-46fb-8977-025f0f0123b7-kube-api-access-4bjv2\") pod \"maas-api-revoke-sub-8b8ead43-jf5d8\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.382647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.382582    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b8ead43-jf5d8\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.382891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.382668    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bv6x\" (UniqueName: \"kubernetes.io/projected/0589a5ea-bf21-48fd-8534-8e6866a2f88a-kube-api-access-2bv6x\") pod \"maas-api-revoke-sub-daf0b9ad-cjq9w\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.382891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.382699    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf0b9ad-cjq9w\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.382891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.382858    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf0b9ad-cjq9w\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.383059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.382925    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b8ead43-jf5d8\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.383769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.383740    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b8ead43-jf5d8\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.385308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.385283    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b8ead43-jf5d8\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.391859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.391824    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjv2\" (UniqueName: \"kubernetes.io/projected/ff68e187-075b-46fb-8977-025f0f0123b7-kube-api-access-4bjv2\") pod \"maas-api-revoke-sub-8b8ead43-jf5d8\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.484154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.483937    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bv6x\" (UniqueName: \"kubernetes.io/projected/0589a5ea-bf21-48fd-8534-8e6866a2f88a-kube-api-access-2bv6x\") pod \"maas-api-revoke-sub-daf0b9ad-cjq9w\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.484154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.484002    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf0b9ad-cjq9w\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.484538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.484454    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf0b9ad-cjq9w\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.486177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.486153    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf0b9ad-cjq9w\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.489158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.489111    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf0b9ad-cjq9w\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.494995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.494964    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bv6x\" (UniqueName: \"kubernetes.io/projected/0589a5ea-bf21-48fd-8534-8e6866a2f88a-kube-api-access-2bv6x\") pod \"maas-api-revoke-sub-daf0b9ad-cjq9w\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.516788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.516752    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:02.592032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.591979    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:02.692297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.690155    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" event={"ID":"eb216de6-36b1-4280-b763-fb951b3bb9d6","Type":"ContainerStarted","Data":"27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9"}
Nov 03 13:21:02.692297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.690273    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" event={"ID":"eb216de6-36b1-4280-b763-fb951b3bb9d6","Type":"ContainerStarted","Data":"a752f1f69b5bec658596900f506b9d0e9cf3109806b392a7e31ab8ca1c5f37f3"}
Nov 03 13:21:02.696240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.694935    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" event={"ID":"3fd652cb-d578-4f95-b538-ed75795b5ddd","Type":"ContainerStarted","Data":"2e16ab8aee9d12a55b986cb41472b3c866875baa1a7945669d4594fff9c49b08"}
Nov 03 13:21:02.696240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.695013    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" event={"ID":"3fd652cb-d578-4f95-b538-ed75795b5ddd","Type":"ContainerStarted","Data":"1dcb92e2f68e2fa4a873bcbd624a0082ddeb75a47edc74aeeb79eb00b8e2a164"}
Nov 03 13:21:02.707514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.707423    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:02.761391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.760334    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" podStartSLOduration=1.760312501 podStartE2EDuration="1.760312501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:02.744072149 +0000 UTC m=+703.821837191" watchObservedRunningTime="2026-11-03 13:21:02.760312501 +0000 UTC m=+703.838077529"
Nov 03 13:21:02.761391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.760735    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" podStartSLOduration=1.760722342 podStartE2EDuration="1.760722342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:02.712544039 +0000 UTC m=+703.790309066" watchObservedRunningTime="2026-11-03 13:21:02.760722342 +0000 UTC m=+703.838487370"
Nov 03 13:21:02.782264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.771983    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:02.876437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.874282    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"]
Nov 03 13:21:02.889326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.889269    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:02.994283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:02.994258    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"]
Nov 03 13:21:03.704239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.703332    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 13:21:03.704239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.703382    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 13:21:03.709170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.708594    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w" event={"ID":"0589a5ea-bf21-48fd-8534-8e6866a2f88a","Type":"ContainerStarted","Data":"d65c7e3b6bc3f7d3b0a1b98a24d541e66d7084639177115b2bcc1fd5c93b8503"}
Nov 03 13:21:03.709170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.708650    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w" event={"ID":"0589a5ea-bf21-48fd-8534-8e6866a2f88a","Type":"ContainerStarted","Data":"13f0868793575c73aa84c8e46be5793b0ab69bab635ff21125488faf9cd880c5"}
Nov 03 13:21:03.710279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.710241    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8" event={"ID":"ff68e187-075b-46fb-8977-025f0f0123b7","Type":"ContainerStarted","Data":"73b916646ef3a4a99812a4a57891c1d1e9ef47cbb883978bf83547f6eb9fe4f3"}
Nov 03 13:21:03.710383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.710294    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8" event={"ID":"ff68e187-075b-46fb-8977-025f0f0123b7","Type":"ContainerStarted","Data":"9b4411dfa845d532e61d1ec2a294536ef505436820ae31a111e8f199ee7af43b"}
Nov 03 13:21:03.776822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.776736    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 13:21:03.776822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.776793    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 13:21:03.777977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.777947    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:03.778135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.778084    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8" podStartSLOduration=1.778066696 podStartE2EDuration="1.778066696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:03.774406596 +0000 UTC m=+704.852171624" watchObservedRunningTime="2026-11-03 13:21:03.778066696 +0000 UTC m=+704.855831723"
Nov 03 13:21:03.780053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:03.780017    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w" podStartSLOduration=1.780005587 podStartE2EDuration="1.780005587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:03.738318057 +0000 UTC m=+704.816083085" watchObservedRunningTime="2026-11-03 13:21:03.780005587 +0000 UTC m=+704.857770606"
Nov 03 13:21:04.701718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:04.701690    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 13:21:04.702467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:04.702438    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 13:21:04.703537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:04.703509    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:04.719498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:04.719468    5420 generic.go:356] "Generic (PLEG): container finished" podID="ff68e187-075b-46fb-8977-025f0f0123b7" containerID="73b916646ef3a4a99812a4a57891c1d1e9ef47cbb883978bf83547f6eb9fe4f3" exitCode=0
Nov 03 13:21:04.719935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:04.719552    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8" event={"ID":"ff68e187-075b-46fb-8977-025f0f0123b7","Type":"ContainerDied","Data":"73b916646ef3a4a99812a4a57891c1d1e9ef47cbb883978bf83547f6eb9fe4f3"}
Nov 03 13:21:04.739700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:04.736379    5420 generic.go:356] "Generic (PLEG): container finished" podID="0589a5ea-bf21-48fd-8534-8e6866a2f88a" containerID="d65c7e3b6bc3f7d3b0a1b98a24d541e66d7084639177115b2bcc1fd5c93b8503" exitCode=0
Nov 03 13:21:04.739700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:04.737355    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w" event={"ID":"0589a5ea-bf21-48fd-8534-8e6866a2f88a","Type":"ContainerDied","Data":"d65c7e3b6bc3f7d3b0a1b98a24d541e66d7084639177115b2bcc1fd5c93b8503"}
Nov 03 13:21:04.769710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:04.769670    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 13:21:04.769928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:04.769741    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 13:21:05.418244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:05.417787    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:05.702666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:05.702635    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 13:21:05.702827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:05.702689    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 13:21:05.770989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:05.770827    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 13:21:05.770989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:05.770885    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 13:21:06.017278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.016982    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:06.095730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.095701    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff68e187-075b-46fb-8977-025f0f0123b7-kube-api-access-4bjv2\" (UniqueName: \"kubernetes.io/projected/ff68e187-075b-46fb-8977-025f0f0123b7-kube-api-access-4bjv2\") pod \"ff68e187-075b-46fb-8977-025f0f0123b7\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") "
Nov 03 13:21:06.095889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.095773    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-xks-ca\") pod \"ff68e187-075b-46fb-8977-025f0f0123b7\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") "
Nov 03 13:21:06.095889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.095794    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-openshift-ca\") pod \"ff68e187-075b-46fb-8977-025f0f0123b7\" (UID: \"ff68e187-075b-46fb-8977-025f0f0123b7\") "
Nov 03 13:21:06.096650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.096624    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-openshift-ca" pod "ff68e187-075b-46fb-8977-025f0f0123b7" (UID: "ff68e187-075b-46fb-8977-025f0f0123b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:06.100272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.099299    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-xks-ca" pod "ff68e187-075b-46fb-8977-025f0f0123b7" (UID: "ff68e187-075b-46fb-8977-025f0f0123b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:06.100272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.099344    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff68e187-075b-46fb-8977-025f0f0123b7-kube-api-access-4bjv2" pod "ff68e187-075b-46fb-8977-025f0f0123b7" (UID: "ff68e187-075b-46fb-8977-025f0f0123b7"). InnerVolumeSpecName "kube-api-access-4bjv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:06.118668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.118645    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:06.192896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.192861    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:06.197060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.197033    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0589a5ea-bf21-48fd-8534-8e6866a2f88a-kube-api-access-2bv6x\" (UniqueName: \"kubernetes.io/projected/0589a5ea-bf21-48fd-8534-8e6866a2f88a-kube-api-access-2bv6x\") pod \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") "
Nov 03 13:21:06.197232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.197152    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-xks-ca\") pod \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") "
Nov 03 13:21:06.197232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.197184    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-openshift-ca\") pod \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\" (UID: \"0589a5ea-bf21-48fd-8534-8e6866a2f88a\") "
Nov 03 13:21:06.197673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.197641    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-openshift-ca" pod "0589a5ea-bf21-48fd-8534-8e6866a2f88a" (UID: "0589a5ea-bf21-48fd-8534-8e6866a2f88a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:06.197879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.197858    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:06.197960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.197885    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bjv2\" (UniqueName: \"kubernetes.io/projected/ff68e187-075b-46fb-8977-025f0f0123b7-kube-api-access-4bjv2\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:06.197960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.197902    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:06.197960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.197917    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff68e187-075b-46fb-8977-025f0f0123b7-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:06.199233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.199197    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0589a5ea-bf21-48fd-8534-8e6866a2f88a-kube-api-access-2bv6x" pod "0589a5ea-bf21-48fd-8534-8e6866a2f88a" (UID: "0589a5ea-bf21-48fd-8534-8e6866a2f88a"). InnerVolumeSpecName "kube-api-access-2bv6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:06.199319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.199302    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-xks-ca" pod "0589a5ea-bf21-48fd-8534-8e6866a2f88a" (UID: "0589a5ea-bf21-48fd-8534-8e6866a2f88a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:06.299142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.299061    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0589a5ea-bf21-48fd-8534-8e6866a2f88a-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:06.299142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.299090    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bv6x\" (UniqueName: \"kubernetes.io/projected/0589a5ea-bf21-48fd-8534-8e6866a2f88a-kube-api-access-2bv6x\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:06.681439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.681362    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:21:06.702409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.702377    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 13:21:06.702594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.702438    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 13:21:06.785277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.784804    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 13:21:06.785277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.784903    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 13:21:06.822234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.821820    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w" event={"ID":"0589a5ea-bf21-48fd-8534-8e6866a2f88a","Type":"ContainerDied","Data":"13f0868793575c73aa84c8e46be5793b0ab69bab635ff21125488faf9cd880c5"}
Nov 03 13:21:06.822234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.821887    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13f0868793575c73aa84c8e46be5793b0ab69bab635ff21125488faf9cd880c5"
Nov 03 13:21:06.822234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.821961    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"
Nov 03 13:21:06.842237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.830650    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8" event={"ID":"ff68e187-075b-46fb-8977-025f0f0123b7","Type":"ContainerDied","Data":"9b4411dfa845d532e61d1ec2a294536ef505436820ae31a111e8f199ee7af43b"}
Nov 03 13:21:06.842237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.830712    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b4411dfa845d532e61d1ec2a294536ef505436820ae31a111e8f199ee7af43b"
Nov 03 13:21:06.842237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.830779    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"
Nov 03 13:21:06.956292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:06.953652    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:07.113705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:07.113670    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:07.695238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:07.694831    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:07.704235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:07.701968    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 13:21:07.704235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:07.702011    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 13:21:07.777235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:07.774327    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 13:21:07.777235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:07.774389    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 13:21:08.492236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:08.490906    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:08.707237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:08.706873    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 13:21:08.707237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:08.706934    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 13:21:08.785298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:08.773942    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 13:21:08.785298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:08.774015    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 13:21:09.343236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:09.332589    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:09.343236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:09.340317    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 13:21:09.355237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:09.351689    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:09.389691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:09.389644    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:09.419639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:09.419590    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 13:21:09.555504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:09.555289    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:09.622288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:09.584955    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:09.771177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:09.771128    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 13:21:09.771587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:09.771553    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 13:21:10.410234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:10.404737    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:10.410234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:10.407508    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:10.422112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:10.418059    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:21:10.545102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:10.545069    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:11.323319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:11.323275    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:11.426240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:11.426195    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:11.426739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:11.426719    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:11.427649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:11.427630    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:21:11.961556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:11.961527    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:12.276028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:12.275939    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:12.420169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:12.420142    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:12.610032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:12.609949    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:13.400128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:13.400096    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:14.092827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:14.092794    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:15.005845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:15.005809    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:15.186896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:15.186859    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:15.924091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:15.924057    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:16.356293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.356253    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:16.387362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.387331    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"]
Nov 03 13:21:16.387899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.387886    5420 cpu_manager.go:414] "containerMap: removing container" podUID="ff68e187-075b-46fb-8977-025f0f0123b7" containerName="revoke-keys"
Nov 03 13:21:16.387941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.387901    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff68e187-075b-46fb-8977-025f0f0123b7" containerName="revoke-keys"
Nov 03 13:21:16.387941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.387919    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0589a5ea-bf21-48fd-8534-8e6866a2f88a" containerName="revoke-keys"
Nov 03 13:21:16.387941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.387925    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0589a5ea-bf21-48fd-8534-8e6866a2f88a" containerName="revoke-keys"
Nov 03 13:21:16.388041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.388037    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff68e187-075b-46fb-8977-025f0f0123b7" containerName="revoke-keys"
Nov 03 13:21:16.388079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.388048    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0589a5ea-bf21-48fd-8534-8e6866a2f88a" containerName="revoke-keys"
Nov 03 13:21:16.392922 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.392903    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:16.398430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.398408    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7c192b\""
Nov 03 13:21:16.400024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.399984    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7c192b-dockercfg-4h8tz\""
Nov 03 13:21:16.401913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.401886    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"]
Nov 03 13:21:16.415793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.415762    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"]
Nov 03 13:21:16.420862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.420837    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:16.428441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.428410    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"]
Nov 03 13:21:16.482157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.482120    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cde7e-d888-42b2-ba6b-73d80875a054-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf\" (UID: \"f83cde7e-d888-42b2-ba6b-73d80875a054\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:16.482377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.482260    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snq5x\" (UniqueName: \"kubernetes.io/projected/f83cde7e-d888-42b2-ba6b-73d80875a054-kube-api-access-snq5x\") pod \"payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf\" (UID: \"f83cde7e-d888-42b2-ba6b-73d80875a054\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:16.569236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.557583    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:16.586530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.583840    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cde7e-d888-42b2-ba6b-73d80875a054-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf\" (UID: \"f83cde7e-d888-42b2-ba6b-73d80875a054\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:16.586530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.583908    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snq5x\" (UniqueName: \"kubernetes.io/projected/f83cde7e-d888-42b2-ba6b-73d80875a054-kube-api-access-snq5x\") pod \"payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf\" (UID: \"f83cde7e-d888-42b2-ba6b-73d80875a054\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:16.586530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.583982    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2\" (UID: \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:16.586530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.584047    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dvv\" (UniqueName: \"kubernetes.io/projected/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-kube-api-access-89dvv\") pod \"payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2\" (UID: \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:16.586530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.585072    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cde7e-d888-42b2-ba6b-73d80875a054-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf\" (UID: \"f83cde7e-d888-42b2-ba6b-73d80875a054\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:16.611535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.611470    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snq5x\" (UniqueName: \"kubernetes.io/projected/f83cde7e-d888-42b2-ba6b-73d80875a054-kube-api-access-snq5x\") pod \"payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf\" (UID: \"f83cde7e-d888-42b2-ba6b-73d80875a054\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:16.685447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.685413    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2\" (UID: \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:16.685629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.685468    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dvv\" (UniqueName: \"kubernetes.io/projected/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-kube-api-access-89dvv\") pod \"payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2\" (UID: \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:16.686088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.686070    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2\" (UID: \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:16.696702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.696680    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dvv\" (UniqueName: \"kubernetes.io/projected/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-kube-api-access-89dvv\") pod \"payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2\" (UID: \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:16.716837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.716799    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:16.749443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.749413    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:16.897254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.897147    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"]
Nov 03 13:21:16.903330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.903307    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:16.936786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.936755    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"]
Nov 03 13:21:16.992759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.992719    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:16.992935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.992856    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:16.992935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.992920    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:16.993042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.993022    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:16.993153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.993136    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lkgz\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-kube-api-access-6lkgz\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:16.993265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.993202    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:16.993325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.993293    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:16.993485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.993464    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:16.993550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:16.993510    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.041908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.041879    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"]
Nov 03 13:21:17.042768 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:17.042739    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83cde7e_d888_42b2_ba6b_73d80875a054.slice/crio-81565f3d0b1111522e812a5e4e8ea6bb2d73c734671c45161d13b88da78ba86b WatchSource:0}: Error finding container 81565f3d0b1111522e812a5e4e8ea6bb2d73c734671c45161d13b88da78ba86b: Status 404 returned error can't find the container with id 81565f3d0b1111522e812a5e4e8ea6bb2d73c734671c45161d13b88da78ba86b
Nov 03 13:21:17.096050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.095963    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.096370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.096324    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.096614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.096576    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.096915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.096886    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lkgz\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-kube-api-access-6lkgz\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.097034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.096985    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.097100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.097062    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.097160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.097113    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.097160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.097148    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.097375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.097192    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.098307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.097671    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.102728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.102695    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.102911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.102888    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.103371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.103344    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.103632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.103615    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.105189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.105137    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.112421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.112394    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.113717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.113663    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.119317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.119276    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"]
Nov 03 13:21:17.124809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.124782    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lkgz\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-kube-api-access-6lkgz\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.155851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.155771    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"]
Nov 03 13:21:17.156462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.156439    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:17.159938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.159917    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.167958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.167935    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:21:17.168425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.168405    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:21:17.168887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.168869    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:21:17.169396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.169279    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:21:17.174619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.174596    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"]
Nov 03 13:21:17.220884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.220844    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:17.299168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.299138    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwjv\" (UniqueName: \"kubernetes.io/projected/056d541c-e3b4-44a5-9d8d-74360e94a576-kube-api-access-cqwjv\") pod \"maas-api-revoke-sub-dfff674b-z9clp\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.299372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.299279    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfff674b-z9clp\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.299372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.299334    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfff674b-z9clp\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.328282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.328250    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:17.400369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.400331    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwjv\" (UniqueName: \"kubernetes.io/projected/056d541c-e3b4-44a5-9d8d-74360e94a576-kube-api-access-cqwjv\") pod \"maas-api-revoke-sub-dfff674b-z9clp\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.400826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.400434    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfff674b-z9clp\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.400826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.400472    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfff674b-z9clp\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.401330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.401303    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfff674b-z9clp\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.403296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.403274    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfff674b-z9clp\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.408765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.408711    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwjv\" (UniqueName: \"kubernetes.io/projected/056d541c-e3b4-44a5-9d8d-74360e94a576-kube-api-access-cqwjv\") pod \"maas-api-revoke-sub-dfff674b-z9clp\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.481807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.481634    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:17.517263 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:17.517181    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca7984a_88de_4f3e_8b32_ca9f9e15a649.slice/crio-4a00b5f17d4377a2e8e0654d3bf288cb9df90be7067ca6b367c579cbe7ced6be WatchSource:0}: Error finding container 4a00b5f17d4377a2e8e0654d3bf288cb9df90be7067ca6b367c579cbe7ced6be: Status 404 returned error can't find the container with id 4a00b5f17d4377a2e8e0654d3bf288cb9df90be7067ca6b367c579cbe7ced6be
Nov 03 13:21:17.518167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.518051    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"]
Nov 03 13:21:17.524745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.524247    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:17.524745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.524387    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:17.524745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.524477    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:17.538456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.538421    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"]
Nov 03 13:21:17.555708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.552067    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.555708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.554908    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"]
Nov 03 13:21:17.704448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.704413    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eee2e40-k6sz8\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.704600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.704458    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz7pw\" (UniqueName: \"kubernetes.io/projected/eef92699-edad-4df1-98c0-ef51e8c07891-kube-api-access-nz7pw\") pod \"maas-api-revoke-sub-6eee2e40-k6sz8\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.704600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.704579    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eee2e40-k6sz8\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.782156 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:17.782120    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056d541c_e3b4_44a5_9d8d_74360e94a576.slice/crio-2efd10e085778973c8f11078335cddcd30f61cbd4371d5d523b4982aa51d4461 WatchSource:0}: Error finding container 2efd10e085778973c8f11078335cddcd30f61cbd4371d5d523b4982aa51d4461: Status 404 returned error can't find the container with id 2efd10e085778973c8f11078335cddcd30f61cbd4371d5d523b4982aa51d4461
Nov 03 13:21:17.782327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.782271    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"]
Nov 03 13:21:17.806620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.806587    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eee2e40-k6sz8\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.806718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.806639    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz7pw\" (UniqueName: \"kubernetes.io/projected/eef92699-edad-4df1-98c0-ef51e8c07891-kube-api-access-nz7pw\") pod \"maas-api-revoke-sub-6eee2e40-k6sz8\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.806718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.806702    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eee2e40-k6sz8\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.807364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.807341    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eee2e40-k6sz8\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.808993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.808969    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eee2e40-k6sz8\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.815176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.815150    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz7pw\" (UniqueName: \"kubernetes.io/projected/eef92699-edad-4df1-98c0-ef51e8c07891-kube-api-access-nz7pw\") pod \"maas-api-revoke-sub-6eee2e40-k6sz8\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:17.882388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:17.882355    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:18.007371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.007269    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf" event={"ID":"f83cde7e-d888-42b2-ba6b-73d80875a054","Type":"ContainerStarted","Data":"42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de"}
Nov 03 13:21:18.007371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.007344    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:18.007371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.007371    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf" event={"ID":"f83cde7e-d888-42b2-ba6b-73d80875a054","Type":"ContainerStarted","Data":"81565f3d0b1111522e812a5e4e8ea6bb2d73c734671c45161d13b88da78ba86b"}
Nov 03 13:21:18.009181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.009158    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2" event={"ID":"56237f7c-9a5e-4437-b33a-38d06e5cc1e2","Type":"ContainerStarted","Data":"4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854"}
Nov 03 13:21:18.009329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.009202    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2" event={"ID":"56237f7c-9a5e-4437-b33a-38d06e5cc1e2","Type":"ContainerStarted","Data":"94f63a87bd3515d88b5c75c2036c908c757c4e6535b07bc337f51551c5c32491"}
Nov 03 13:21:18.009329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.009263    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:18.011180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.011147    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp" event={"ID":"056d541c-e3b4-44a5-9d8d-74360e94a576","Type":"ContainerStarted","Data":"2efd10e085778973c8f11078335cddcd30f61cbd4371d5d523b4982aa51d4461"}
Nov 03 13:21:18.012899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.012879    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" event={"ID":"4ca7984a-88de-4f3e-8b32-ca9f9e15a649","Type":"ContainerStarted","Data":"ea580a6504a2c949bfa62913b2d572d861726456f18ea43ebd1a9146d60e5b13"}
Nov 03 13:21:18.012992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.012920    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" event={"ID":"4ca7984a-88de-4f3e-8b32-ca9f9e15a649","Type":"ContainerStarted","Data":"4a00b5f17d4377a2e8e0654d3bf288cb9df90be7067ca6b367c579cbe7ced6be"}
Nov 03 13:21:18.032654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.032587    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf" podStartSLOduration=2.03256513 podStartE2EDuration="2.03256513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:18.024658069 +0000 UTC m=+719.102423095" watchObservedRunningTime="2026-11-03 13:21:18.03256513 +0000 UTC m=+719.110330157"
Nov 03 13:21:18.058429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.058359    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2" podStartSLOduration=2.058336988 podStartE2EDuration="2.058336988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:18.052509875 +0000 UTC m=+719.130274902" watchObservedRunningTime="2026-11-03 13:21:18.058336988 +0000 UTC m=+719.136102015"
Nov 03 13:21:18.086444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.086392    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podStartSLOduration=2.086377059 podStartE2EDuration="2.086377059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:18.083748021 +0000 UTC m=+719.161513047" watchObservedRunningTime="2026-11-03 13:21:18.086377059 +0000 UTC m=+719.164142085"
Nov 03 13:21:18.112683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.112655    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:18.177281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.176542    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"]
Nov 03 13:21:18.180171 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:18.180129    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef92699_edad_4df1_98c0_ef51e8c07891.slice/crio-32b6ed28e1580e01afcaeb873c5c94bcfe5cda305af9a22bc6bbc03f20acc911 WatchSource:0}: Error finding container 32b6ed28e1580e01afcaeb873c5c94bcfe5cda305af9a22bc6bbc03f20acc911: Status 404 returned error can't find the container with id 32b6ed28e1580e01afcaeb873c5c94bcfe5cda305af9a22bc6bbc03f20acc911
Nov 03 13:21:18.240502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.234583    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:18.241683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.241337    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 13:21:18.241683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:18.241397    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 13:21:19.034606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:19.029494    5420 generic.go:356] "Generic (PLEG): container finished" podID="056d541c-e3b4-44a5-9d8d-74360e94a576" containerID="03ef318fe778be99da0605bb5eb908dc9e59949848b33bf04b6b10acb137cb46" exitCode=0
Nov 03 13:21:19.034606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:19.029850    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp" event={"ID":"056d541c-e3b4-44a5-9d8d-74360e94a576","Type":"ContainerDied","Data":"03ef318fe778be99da0605bb5eb908dc9e59949848b33bf04b6b10acb137cb46"}
Nov 03 13:21:19.051073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:19.050997    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8" event={"ID":"eef92699-edad-4df1-98c0-ef51e8c07891","Type":"ContainerStarted","Data":"d8b37c8c3eebd9a38faa88d3158afa6e8bd987d313c3f3cad316e6e4d9884d4a"}
Nov 03 13:21:19.051282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:19.051098    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8" event={"ID":"eef92699-edad-4df1-98c0-ef51e8c07891","Type":"ContainerStarted","Data":"32b6ed28e1580e01afcaeb873c5c94bcfe5cda305af9a22bc6bbc03f20acc911"}
Nov 03 13:21:19.074800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:19.074752    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8" podStartSLOduration=2.074738763 podStartE2EDuration="2.074738763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:19.071813765 +0000 UTC m=+720.149578793" watchObservedRunningTime="2026-11-03 13:21:19.074738763 +0000 UTC m=+720.152503789"
Nov 03 13:21:19.084274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:19.084189    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:19.241022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:19.240984    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 13:21:19.241348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:19.241320    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 13:21:19.994886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:19.994849    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:20.063154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.063125    5420 generic.go:356] "Generic (PLEG): container finished" podID="eef92699-edad-4df1-98c0-ef51e8c07891" containerID="d8b37c8c3eebd9a38faa88d3158afa6e8bd987d313c3f3cad316e6e4d9884d4a" exitCode=0
Nov 03 13:21:20.063644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.063206    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8" event={"ID":"eef92699-edad-4df1-98c0-ef51e8c07891","Type":"ContainerDied","Data":"d8b37c8c3eebd9a38faa88d3158afa6e8bd987d313c3f3cad316e6e4d9884d4a"}
Nov 03 13:21:20.242241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.239640    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 13:21:20.242241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.239707    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 13:21:20.276023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.275989    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:20.463171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.463138    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-openshift-ca\") pod \"056d541c-e3b4-44a5-9d8d-74360e94a576\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") "
Nov 03 13:21:20.463406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.463206    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056d541c-e3b4-44a5-9d8d-74360e94a576-kube-api-access-cqwjv\" (UniqueName: \"kubernetes.io/projected/056d541c-e3b4-44a5-9d8d-74360e94a576-kube-api-access-cqwjv\") pod \"056d541c-e3b4-44a5-9d8d-74360e94a576\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") "
Nov 03 13:21:20.463406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.463337    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-xks-ca\") pod \"056d541c-e3b4-44a5-9d8d-74360e94a576\" (UID: \"056d541c-e3b4-44a5-9d8d-74360e94a576\") "
Nov 03 13:21:20.463615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.463591    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-openshift-ca" pod "056d541c-e3b4-44a5-9d8d-74360e94a576" (UID: "056d541c-e3b4-44a5-9d8d-74360e94a576"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:20.463901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.463865    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:20.465367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.465338    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-xks-ca" pod "056d541c-e3b4-44a5-9d8d-74360e94a576" (UID: "056d541c-e3b4-44a5-9d8d-74360e94a576"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:20.465471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.465371    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056d541c-e3b4-44a5-9d8d-74360e94a576-kube-api-access-cqwjv" pod "056d541c-e3b4-44a5-9d8d-74360e94a576" (UID: "056d541c-e3b4-44a5-9d8d-74360e94a576"). InnerVolumeSpecName "kube-api-access-cqwjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:20.564884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.564797    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056d541c-e3b4-44a5-9d8d-74360e94a576-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:20.564884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.564828    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqwjv\" (UniqueName: \"kubernetes.io/projected/056d541c-e3b4-44a5-9d8d-74360e94a576-kube-api-access-cqwjv\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:20.818356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.818276    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"]
Nov 03 13:21:20.818911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.818893    5420 cpu_manager.go:414] "containerMap: removing container" podUID="056d541c-e3b4-44a5-9d8d-74360e94a576" containerName="revoke-keys"
Nov 03 13:21:20.819002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.818912    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056d541c-e3b4-44a5-9d8d-74360e94a576" containerName="revoke-keys"
Nov 03 13:21:20.819100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.819088    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="056d541c-e3b4-44a5-9d8d-74360e94a576" containerName="revoke-keys"
Nov 03 13:21:20.838235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.835713    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:20.851618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.851554    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"]
Nov 03 13:21:20.946038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.945998    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:20.970105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.970075    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9q26\" (UniqueName: \"kubernetes.io/projected/45c204ac-92cf-4322-8dff-c8c1ae3c5100-kube-api-access-j9q26\") pod \"maas-api-revoke-sub-a39b1980-2h2xm\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:20.970302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.970116    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a39b1980-2h2xm\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:20.970302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:20.970267    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a39b1980-2h2xm\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:21.073969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.073866    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9q26\" (UniqueName: \"kubernetes.io/projected/45c204ac-92cf-4322-8dff-c8c1ae3c5100-kube-api-access-j9q26\") pod \"maas-api-revoke-sub-a39b1980-2h2xm\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:21.073969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.073925    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a39b1980-2h2xm\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:21.074522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.073997    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a39b1980-2h2xm\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:21.075638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.075608    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a39b1980-2h2xm\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:21.077553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.077526    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a39b1980-2h2xm\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:21.092273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.086558    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"
Nov 03 13:21:21.092273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.086977    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9q26\" (UniqueName: \"kubernetes.io/projected/45c204ac-92cf-4322-8dff-c8c1ae3c5100-kube-api-access-j9q26\") pod \"maas-api-revoke-sub-a39b1980-2h2xm\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:21.100757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.100714    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp" event={"ID":"056d541c-e3b4-44a5-9d8d-74360e94a576","Type":"ContainerDied","Data":"2efd10e085778973c8f11078335cddcd30f61cbd4371d5d523b4982aa51d4461"}
Nov 03 13:21:21.100757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.100760    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2efd10e085778973c8f11078335cddcd30f61cbd4371d5d523b4982aa51d4461"
Nov 03 13:21:21.101422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.101364    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:21.173057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.173021    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:21.237236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.224624    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 13:21:21.237236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.224686    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 13:21:21.373082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.372747    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:21.481335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.481301    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-openshift-ca\") pod \"eef92699-edad-4df1-98c0-ef51e8c07891\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") "
Nov 03 13:21:21.481528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.481389    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef92699-edad-4df1-98c0-ef51e8c07891-kube-api-access-nz7pw\" (UniqueName: \"kubernetes.io/projected/eef92699-edad-4df1-98c0-ef51e8c07891-kube-api-access-nz7pw\") pod \"eef92699-edad-4df1-98c0-ef51e8c07891\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") "
Nov 03 13:21:21.481598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.481563    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-xks-ca\") pod \"eef92699-edad-4df1-98c0-ef51e8c07891\" (UID: \"eef92699-edad-4df1-98c0-ef51e8c07891\") "
Nov 03 13:21:21.481731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.481702    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-openshift-ca" pod "eef92699-edad-4df1-98c0-ef51e8c07891" (UID: "eef92699-edad-4df1-98c0-ef51e8c07891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:21.482089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.482065    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:21.483611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.483590    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-xks-ca" pod "eef92699-edad-4df1-98c0-ef51e8c07891" (UID: "eef92699-edad-4df1-98c0-ef51e8c07891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:21.483611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.483593    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef92699-edad-4df1-98c0-ef51e8c07891-kube-api-access-nz7pw" pod "eef92699-edad-4df1-98c0-ef51e8c07891" (UID: "eef92699-edad-4df1-98c0-ef51e8c07891"). InnerVolumeSpecName "kube-api-access-nz7pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:21.500488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.500461    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"]
Nov 03 13:21:21.501095 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:21.501069    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c204ac_92cf_4322_8dff_c8c1ae3c5100.slice/crio-b17e104bc656d28bad34488376ecc87ccfe0e7fc401aa1a1202f7b8237630270 WatchSource:0}: Error finding container b17e104bc656d28bad34488376ecc87ccfe0e7fc401aa1a1202f7b8237630270: Status 404 returned error can't find the container with id b17e104bc656d28bad34488376ecc87ccfe0e7fc401aa1a1202f7b8237630270
Nov 03 13:21:21.583965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.583936    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef92699-edad-4df1-98c0-ef51e8c07891-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:21.583965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.583963    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz7pw\" (UniqueName: \"kubernetes.io/projected/eef92699-edad-4df1-98c0-ef51e8c07891-kube-api-access-nz7pw\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:21.950408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:21.950363    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:22.104201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:22.098975    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"
Nov 03 13:21:22.104201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:22.099083    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8" event={"ID":"eef92699-edad-4df1-98c0-ef51e8c07891","Type":"ContainerDied","Data":"32b6ed28e1580e01afcaeb873c5c94bcfe5cda305af9a22bc6bbc03f20acc911"}
Nov 03 13:21:22.104201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:22.099128    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b6ed28e1580e01afcaeb873c5c94bcfe5cda305af9a22bc6bbc03f20acc911"
Nov 03 13:21:22.104201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:22.101436    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm" event={"ID":"45c204ac-92cf-4322-8dff-c8c1ae3c5100","Type":"ContainerStarted","Data":"984e5daca7299738db7098243f2419cbcda162deea16b93d2ef2c0a34e2d8925"}
Nov 03 13:21:22.104201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:22.101485    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm" event={"ID":"45c204ac-92cf-4322-8dff-c8c1ae3c5100","Type":"ContainerStarted","Data":"b17e104bc656d28bad34488376ecc87ccfe0e7fc401aa1a1202f7b8237630270"}
Nov 03 13:21:22.141076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:22.141011    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm" podStartSLOduration=2.140990691 podStartE2EDuration="2.140990691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:22.130574031 +0000 UTC m=+723.208339057" watchObservedRunningTime="2026-11-03 13:21:22.140990691 +0000 UTC m=+723.218755727"
Nov 03 13:21:22.225234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:22.224644    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 13:21:22.225234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:22.224721    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 13:21:22.844235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:22.844138    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:23.008798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:23.008707    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:23.191237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:23.191137    5420 generic.go:356] "Generic (PLEG): container finished" podID="45c204ac-92cf-4322-8dff-c8c1ae3c5100" containerID="984e5daca7299738db7098243f2419cbcda162deea16b93d2ef2c0a34e2d8925" exitCode=0
Nov 03 13:21:23.191753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:23.191342    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm" event={"ID":"45c204ac-92cf-4322-8dff-c8c1ae3c5100","Type":"ContainerDied","Data":"984e5daca7299738db7098243f2419cbcda162deea16b93d2ef2c0a34e2d8925"}
Nov 03 13:21:23.295298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:23.280284    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 13:21:23.312855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:23.312532    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 13:21:23.409343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:23.409294    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:23.497265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:23.491089    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:23.982237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:23.980262    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:23.992136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:23.992098    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:24.221733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.221693    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 13:21:24.224827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.224784    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 13:21:24.329422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.329317    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:24.355358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.355312    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:24.365196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.365151    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 13:21:24.390118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.389122    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:24.412510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.412446    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:24.492361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.492314    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:24.493022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.492998    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:21:24.538238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.535365    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:24.663319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:24.662319    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded"
Nov 03 13:21:24.902236 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:21:24.901003    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:21:25.247410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.247372    5420 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 13:21:25.274918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.248557    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 13:21:25.379231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.376383    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:25.384869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.384828    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:25.385302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.385277    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:21:25.396538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.395145    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:21:25.408180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.408156    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:21:25.459250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.458551    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-openshift-ca\") pod \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") "
Nov 03 13:21:25.459250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.458623    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c204ac-92cf-4322-8dff-c8c1ae3c5100-kube-api-access-j9q26\" (UniqueName: \"kubernetes.io/projected/45c204ac-92cf-4322-8dff-c8c1ae3c5100-kube-api-access-j9q26\") pod \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") "
Nov 03 13:21:25.459250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.458672    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-xks-ca\") pod \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\" (UID: \"45c204ac-92cf-4322-8dff-c8c1ae3c5100\") "
Nov 03 13:21:25.460877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.460844    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-openshift-ca" pod "45c204ac-92cf-4322-8dff-c8c1ae3c5100" (UID: "45c204ac-92cf-4322-8dff-c8c1ae3c5100"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:25.463579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.463544    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-xks-ca" pod "45c204ac-92cf-4322-8dff-c8c1ae3c5100" (UID: "45c204ac-92cf-4322-8dff-c8c1ae3c5100"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:25.463696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.463657    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c204ac-92cf-4322-8dff-c8c1ae3c5100-kube-api-access-j9q26" pod "45c204ac-92cf-4322-8dff-c8c1ae3c5100" (UID: "45c204ac-92cf-4322-8dff-c8c1ae3c5100"). InnerVolumeSpecName "kube-api-access-j9q26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:25.560826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.560790    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:25.560826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.560822    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9q26\" (UniqueName: \"kubernetes.io/projected/45c204ac-92cf-4322-8dff-c8c1ae3c5100-kube-api-access-j9q26\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:25.560826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.560832    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45c204ac-92cf-4322-8dff-c8c1ae3c5100-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:25.594979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:25.594949    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:26.384262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:26.384142    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"
Nov 03 13:21:26.384262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:26.384165    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm" event={"ID":"45c204ac-92cf-4322-8dff-c8c1ae3c5100","Type":"ContainerDied","Data":"b17e104bc656d28bad34488376ecc87ccfe0e7fc401aa1a1202f7b8237630270"}
Nov 03 13:21:26.384262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:26.384247    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17e104bc656d28bad34488376ecc87ccfe0e7fc401aa1a1202f7b8237630270"
Nov 03 13:21:26.450236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:26.448663    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:26.541929 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:26.541896    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:26.542660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:26.542633    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:26.543848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:26.543827    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:21:27.162759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:27.162731    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:27.916078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:27.916043    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:28.699738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:28.699711    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:29.053691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.053663    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:21:29.054079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.053734    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:21:29.437045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.436968    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"]
Nov 03 13:21:29.437566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.437552    5420 cpu_manager.go:414] "containerMap: removing container" podUID="eef92699-edad-4df1-98c0-ef51e8c07891" containerName="revoke-keys"
Nov 03 13:21:29.437606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.437567    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef92699-edad-4df1-98c0-ef51e8c07891" containerName="revoke-keys"
Nov 03 13:21:29.437606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.437579    5420 cpu_manager.go:414] "containerMap: removing container" podUID="45c204ac-92cf-4322-8dff-c8c1ae3c5100" containerName="revoke-keys"
Nov 03 13:21:29.437606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.437584    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c204ac-92cf-4322-8dff-c8c1ae3c5100" containerName="revoke-keys"
Nov 03 13:21:29.437702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.437673    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef92699-edad-4df1-98c0-ef51e8c07891" containerName="revoke-keys"
Nov 03 13:21:29.437702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.437683    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c204ac-92cf-4322-8dff-c8c1ae3c5100" containerName="revoke-keys"
Nov 03 13:21:29.441493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.441471    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.445434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.445372    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:21:29.445558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.445373    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:21:29.446786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.446763    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:21:29.446786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.446767    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:21:29.447660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.447635    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"]
Nov 03 13:21:29.510168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.510132    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-444ae124-7v2gn\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.510382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.510358    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj67x\" (UniqueName: \"kubernetes.io/projected/02cd03f8-4f08-437b-8f56-1bc54b840aff-kube-api-access-fj67x\") pod \"maas-api-revoke-sub-444ae124-7v2gn\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.510439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.510396    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-444ae124-7v2gn\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.514237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.514191    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:29.611323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.611289    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj67x\" (UniqueName: \"kubernetes.io/projected/02cd03f8-4f08-437b-8f56-1bc54b840aff-kube-api-access-fj67x\") pod \"maas-api-revoke-sub-444ae124-7v2gn\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.611518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.611335    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-444ae124-7v2gn\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.611518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.611454    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-444ae124-7v2gn\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.612174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.612148    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-444ae124-7v2gn\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.613972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.613948    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-444ae124-7v2gn\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.620901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.620868    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj67x\" (UniqueName: \"kubernetes.io/projected/02cd03f8-4f08-437b-8f56-1bc54b840aff-kube-api-access-fj67x\") pod \"maas-api-revoke-sub-444ae124-7v2gn\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.725549 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.725515    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"]
Nov 03 13:21:29.729465 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.729447    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:29.737456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.737431    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"]
Nov 03 13:21:29.758444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.758412    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:29.813497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.813456    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jr9\" (UniqueName: \"kubernetes.io/projected/65a1d702-1b69-48fb-ae4b-8fd168912f17-kube-api-access-z9jr9\") pod \"maas-api-revoke-sub-283868ba-nkpmc\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:29.813613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.813596    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-283868ba-nkpmc\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:29.813686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.813662    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-283868ba-nkpmc\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:29.915470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.915432    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-283868ba-nkpmc\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:29.915652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.915618    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jr9\" (UniqueName: \"kubernetes.io/projected/65a1d702-1b69-48fb-ae4b-8fd168912f17-kube-api-access-z9jr9\") pod \"maas-api-revoke-sub-283868ba-nkpmc\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:29.915752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.915734    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-283868ba-nkpmc\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:29.916184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.916165    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-283868ba-nkpmc\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:29.918006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.917987    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-283868ba-nkpmc\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:29.923952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:29.923932    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jr9\" (UniqueName: \"kubernetes.io/projected/65a1d702-1b69-48fb-ae4b-8fd168912f17-kube-api-access-z9jr9\") pod \"maas-api-revoke-sub-283868ba-nkpmc\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:30.039544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:30.039512    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"]
Nov 03 13:21:30.039659 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:30.039607    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cd03f8_4f08_437b_8f56_1bc54b840aff.slice/crio-47f9db1b0ec6e790e49336d252f894d3d20ba4a5eb623cfa670566cda8689fec WatchSource:0}: Error finding container 47f9db1b0ec6e790e49336d252f894d3d20ba4a5eb623cfa670566cda8689fec: Status 404 returned error can't find the container with id 47f9db1b0ec6e790e49336d252f894d3d20ba4a5eb623cfa670566cda8689fec
Nov 03 13:21:30.047139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:30.047106    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:30.098583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:30.098553    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:30.333521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:30.333492    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"]
Nov 03 13:21:30.334272 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:30.334205    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a1d702_1b69_48fb_ae4b_8fd168912f17.slice/crio-891098cb03f9ef0262514f1d7905832c900a979c4dc0a6acee97e95a068b03f2 WatchSource:0}: Error finding container 891098cb03f9ef0262514f1d7905832c900a979c4dc0a6acee97e95a068b03f2: Status 404 returned error can't find the container with id 891098cb03f9ef0262514f1d7905832c900a979c4dc0a6acee97e95a068b03f2
Nov 03 13:21:30.419024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:30.418991    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn" event={"ID":"02cd03f8-4f08-437b-8f56-1bc54b840aff","Type":"ContainerStarted","Data":"47f9db1b0ec6e790e49336d252f894d3d20ba4a5eb623cfa670566cda8689fec"}
Nov 03 13:21:30.420357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:30.420331    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc" event={"ID":"65a1d702-1b69-48fb-ae4b-8fd168912f17","Type":"ContainerStarted","Data":"891098cb03f9ef0262514f1d7905832c900a979c4dc0a6acee97e95a068b03f2"}
Nov 03 13:21:30.724370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:30.724294    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:31.035858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.035825    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"]
Nov 03 13:21:31.040329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.040305    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.046859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.046833    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"]
Nov 03 13:21:31.131290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.131242    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b554560c-67gh6\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.131752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.131500    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b554560c-67gh6\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.131752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.131572    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dh4\" (UniqueName: \"kubernetes.io/projected/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-kube-api-access-n5dh4\") pod \"maas-api-revoke-sub-b554560c-67gh6\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.233408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.233373    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b554560c-67gh6\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.233539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.233524    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b554560c-67gh6\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.233584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.233552    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dh4\" (UniqueName: \"kubernetes.io/projected/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-kube-api-access-n5dh4\") pod \"maas-api-revoke-sub-b554560c-67gh6\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.234120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.234079    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b554560c-67gh6\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.235890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.235871    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b554560c-67gh6\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.242491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.242472    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dh4\" (UniqueName: \"kubernetes.io/projected/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-kube-api-access-n5dh4\") pod \"maas-api-revoke-sub-b554560c-67gh6\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.358114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.358080    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:31.440743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.440670    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:31.456545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.456507    5420 generic.go:356] "Generic (PLEG): container finished" podID="02cd03f8-4f08-437b-8f56-1bc54b840aff" containerID="3c5b5369727d4be99307c9f0a1872eef5b8ef0b7a0072c92aa09c0661335ee43" exitCode=0
Nov 03 13:21:31.456684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.456618    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn" event={"ID":"02cd03f8-4f08-437b-8f56-1bc54b840aff","Type":"ContainerDied","Data":"3c5b5369727d4be99307c9f0a1872eef5b8ef0b7a0072c92aa09c0661335ee43"}
Nov 03 13:21:31.459161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.459133    5420 generic.go:356] "Generic (PLEG): container finished" podID="65a1d702-1b69-48fb-ae4b-8fd168912f17" containerID="b7b958dff4bc61b1431eaa5645e013a5a11f5c8682a0dac3c6daf11964f39bbb" exitCode=0
Nov 03 13:21:31.459323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.459251    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc" event={"ID":"65a1d702-1b69-48fb-ae4b-8fd168912f17","Type":"ContainerDied","Data":"b7b958dff4bc61b1431eaa5645e013a5a11f5c8682a0dac3c6daf11964f39bbb"}
Nov 03 13:21:31.682544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.682510    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"]
Nov 03 13:21:31.683691 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:31.683646    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a053c9_9607_466c_8a69_b0c2f64ac4f0.slice/crio-5d708f37d42d82f7bdfab555116da51449c6031cd0abf6295f0fe551893661ad WatchSource:0}: Error finding container 5d708f37d42d82f7bdfab555116da51449c6031cd0abf6295f0fe551893661ad: Status 404 returned error can't find the container with id 5d708f37d42d82f7bdfab555116da51449c6031cd0abf6295f0fe551893661ad
Nov 03 13:21:31.992003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:31.991869    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:32.467749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.467714    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6" event={"ID":"f5a053c9-9607-466c-8a69-b0c2f64ac4f0","Type":"ContainerStarted","Data":"f245f63e96fdc25b8e0644df9dac754117719a93433bf5776d65156a4fb2f0a6"}
Nov 03 13:21:32.468205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.467776    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6" event={"ID":"f5a053c9-9607-466c-8a69-b0c2f64ac4f0","Type":"ContainerStarted","Data":"5d708f37d42d82f7bdfab555116da51449c6031cd0abf6295f0fe551893661ad"}
Nov 03 13:21:32.488035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.487972    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6" podStartSLOduration=1.487955918 podStartE2EDuration="1.487955918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:32.483991896 +0000 UTC m=+733.561756936" watchObservedRunningTime="2026-11-03 13:21:32.487955918 +0000 UTC m=+733.565720944"
Nov 03 13:21:32.633853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.633676    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:32.655285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.654912    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:32.690917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.690888    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:32.755903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.755824    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-openshift-ca\") pod \"65a1d702-1b69-48fb-ae4b-8fd168912f17\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") "
Nov 03 13:21:32.755903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.755879    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a1d702-1b69-48fb-ae4b-8fd168912f17-kube-api-access-z9jr9\" (UniqueName: \"kubernetes.io/projected/65a1d702-1b69-48fb-ae4b-8fd168912f17-kube-api-access-z9jr9\") pod \"65a1d702-1b69-48fb-ae4b-8fd168912f17\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") "
Nov 03 13:21:32.756120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.755910    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cd03f8-4f08-437b-8f56-1bc54b840aff-kube-api-access-fj67x\" (UniqueName: \"kubernetes.io/projected/02cd03f8-4f08-437b-8f56-1bc54b840aff-kube-api-access-fj67x\") pod \"02cd03f8-4f08-437b-8f56-1bc54b840aff\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") "
Nov 03 13:21:32.756120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.756063    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-openshift-ca\") pod \"02cd03f8-4f08-437b-8f56-1bc54b840aff\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") "
Nov 03 13:21:32.756120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.756095    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-xks-ca\") pod \"02cd03f8-4f08-437b-8f56-1bc54b840aff\" (UID: \"02cd03f8-4f08-437b-8f56-1bc54b840aff\") "
Nov 03 13:21:32.756291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.756133    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-xks-ca\") pod \"65a1d702-1b69-48fb-ae4b-8fd168912f17\" (UID: \"65a1d702-1b69-48fb-ae4b-8fd168912f17\") "
Nov 03 13:21:32.756291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.756239    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-openshift-ca" pod "65a1d702-1b69-48fb-ae4b-8fd168912f17" (UID: "65a1d702-1b69-48fb-ae4b-8fd168912f17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:32.756483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.756456    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-openshift-ca" pod "02cd03f8-4f08-437b-8f56-1bc54b840aff" (UID: "02cd03f8-4f08-437b-8f56-1bc54b840aff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:32.756760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.756740    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:32.756838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.756766    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:32.758354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.758333    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cd03f8-4f08-437b-8f56-1bc54b840aff-kube-api-access-fj67x" pod "02cd03f8-4f08-437b-8f56-1bc54b840aff" (UID: "02cd03f8-4f08-437b-8f56-1bc54b840aff"). InnerVolumeSpecName "kube-api-access-fj67x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:32.758650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.758623    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a1d702-1b69-48fb-ae4b-8fd168912f17-kube-api-access-z9jr9" pod "65a1d702-1b69-48fb-ae4b-8fd168912f17" (UID: "65a1d702-1b69-48fb-ae4b-8fd168912f17"). InnerVolumeSpecName "kube-api-access-z9jr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:32.758749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.758633    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-xks-ca" pod "02cd03f8-4f08-437b-8f56-1bc54b840aff" (UID: "02cd03f8-4f08-437b-8f56-1bc54b840aff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:32.758813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.758778    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-xks-ca" pod "65a1d702-1b69-48fb-ae4b-8fd168912f17" (UID: "65a1d702-1b69-48fb-ae4b-8fd168912f17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:32.858245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.858177    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9jr9\" (UniqueName: \"kubernetes.io/projected/65a1d702-1b69-48fb-ae4b-8fd168912f17-kube-api-access-z9jr9\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:32.858245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.858253    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj67x\" (UniqueName: \"kubernetes.io/projected/02cd03f8-4f08-437b-8f56-1bc54b840aff-kube-api-access-fj67x\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:32.858453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.858266    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cd03f8-4f08-437b-8f56-1bc54b840aff-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:32.858453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:32.858276    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a1d702-1b69-48fb-ae4b-8fd168912f17-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:33.298674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.298644    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:33.476490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.476460    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"
Nov 03 13:21:33.476976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.476465    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn" event={"ID":"02cd03f8-4f08-437b-8f56-1bc54b840aff","Type":"ContainerDied","Data":"47f9db1b0ec6e790e49336d252f894d3d20ba4a5eb623cfa670566cda8689fec"}
Nov 03 13:21:33.476976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.476583    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47f9db1b0ec6e790e49336d252f894d3d20ba4a5eb623cfa670566cda8689fec"
Nov 03 13:21:33.477914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.477896    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"
Nov 03 13:21:33.478024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.477930    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc" event={"ID":"65a1d702-1b69-48fb-ae4b-8fd168912f17","Type":"ContainerDied","Data":"891098cb03f9ef0262514f1d7905832c900a979c4dc0a6acee97e95a068b03f2"}
Nov 03 13:21:33.478024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.477971    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="891098cb03f9ef0262514f1d7905832c900a979c4dc0a6acee97e95a068b03f2"
Nov 03 13:21:33.479789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.479763    5420 generic.go:356] "Generic (PLEG): container finished" podID="f5a053c9-9607-466c-8a69-b0c2f64ac4f0" containerID="f245f63e96fdc25b8e0644df9dac754117719a93433bf5776d65156a4fb2f0a6" exitCode=0
Nov 03 13:21:33.479890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.479802    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6" event={"ID":"f5a053c9-9607-466c-8a69-b0c2f64ac4f0","Type":"ContainerDied","Data":"f245f63e96fdc25b8e0644df9dac754117719a93433bf5776d65156a4fb2f0a6"}
Nov 03 13:21:33.919462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:33.919381    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:34.510034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.509987    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:34.656124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.656101    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:34.782016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.781922    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-xks-ca\") pod \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") "
Nov 03 13:21:34.782016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.781987    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-kube-api-access-n5dh4\" (UniqueName: \"kubernetes.io/projected/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-kube-api-access-n5dh4\") pod \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") "
Nov 03 13:21:34.782318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.782298    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-openshift-ca\") pod \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\" (UID: \"f5a053c9-9607-466c-8a69-b0c2f64ac4f0\") "
Nov 03 13:21:34.782665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.782643    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-openshift-ca" pod "f5a053c9-9607-466c-8a69-b0c2f64ac4f0" (UID: "f5a053c9-9607-466c-8a69-b0c2f64ac4f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:21:34.782768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.782757    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:34.784100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.784072    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-xks-ca" pod "f5a053c9-9607-466c-8a69-b0c2f64ac4f0" (UID: "f5a053c9-9607-466c-8a69-b0c2f64ac4f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:21:34.784207 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.784178    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-kube-api-access-n5dh4" pod "f5a053c9-9607-466c-8a69-b0c2f64ac4f0" (UID: "f5a053c9-9607-466c-8a69-b0c2f64ac4f0"). InnerVolumeSpecName "kube-api-access-n5dh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:21:34.883785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.883748    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:34.883785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.883780    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5dh4\" (UniqueName: \"kubernetes.io/projected/f5a053c9-9607-466c-8a69-b0c2f64ac4f0-kube-api-access-n5dh4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:21:34.983722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:34.983688    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:21:35.208760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:35.208728    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:35.497045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:35.497013    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"
Nov 03 13:21:35.497045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:35.497042    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6" event={"ID":"f5a053c9-9607-466c-8a69-b0c2f64ac4f0","Type":"ContainerDied","Data":"5d708f37d42d82f7bdfab555116da51449c6031cd0abf6295f0fe551893661ad"}
Nov 03 13:21:35.497346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:35.497094    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d708f37d42d82f7bdfab555116da51449c6031cd0abf6295f0fe551893661ad"
Nov 03 13:21:35.933009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:35.932916    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:37.356235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:37.355595    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 13:21:37.360231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:37.356760    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:37.377244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:37.376655    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:37.405236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:37.404336    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:37.413235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:37.410468    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:37.454238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:37.452374    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:37.508236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:37.496380    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:38.009237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.007099    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:38.279313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.279169    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:38.366236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.363125    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:38.366236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.363551    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:21:38.384245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.382310    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:38.384245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.382742    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:21:38.409237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.407304    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 03 13:21:38.409237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.408302    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:21:38.413235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.411172    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:38.421234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.417649    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:21:38.472233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.471561    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:38.472233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.471920    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:21:38.521781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.521732    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:38.522690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.522665    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:21:38.754223 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.695358    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:21:38.810243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.810061    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:21:38.831709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.831681    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:21:38.843202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.839693    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc"
Nov 03 13:21:38.857262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.857236    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:21:38.859024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.858991    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:21:38.874766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:38.874737    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:39.783107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:39.783068    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:40.498662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:40.498624    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:40.659160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:40.659118    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:41.162384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.162353    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"]
Nov 03 13:21:41.162919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.162904    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f5a053c9-9607-466c-8a69-b0c2f64ac4f0" containerName="revoke-keys"
Nov 03 13:21:41.162919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.162918    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a053c9-9607-466c-8a69-b0c2f64ac4f0" containerName="revoke-keys"
Nov 03 13:21:41.162994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.162932    5420 cpu_manager.go:414] "containerMap: removing container" podUID="65a1d702-1b69-48fb-ae4b-8fd168912f17" containerName="revoke-keys"
Nov 03 13:21:41.162994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.162938    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a1d702-1b69-48fb-ae4b-8fd168912f17" containerName="revoke-keys"
Nov 03 13:21:41.162994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.162955    5420 cpu_manager.go:414] "containerMap: removing container" podUID="02cd03f8-4f08-437b-8f56-1bc54b840aff" containerName="revoke-keys"
Nov 03 13:21:41.162994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.162960    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cd03f8-4f08-437b-8f56-1bc54b840aff" containerName="revoke-keys"
Nov 03 13:21:41.163119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.163051    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a1d702-1b69-48fb-ae4b-8fd168912f17" containerName="revoke-keys"
Nov 03 13:21:41.163119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.163060    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cd03f8-4f08-437b-8f56-1bc54b840aff" containerName="revoke-keys"
Nov 03 13:21:41.163119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.163070    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a053c9-9607-466c-8a69-b0c2f64ac4f0" containerName="revoke-keys"
Nov 03 13:21:41.167035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.167014    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.173591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.173561    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"]
Nov 03 13:21:41.207141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.207108    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:41.211912 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.211886    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-dshm\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.212047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.211953    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5a7abc-d740-431c-b70d-d0e7190f8833-tls-certs\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.212122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.212102    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-tmp-dir\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.212173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.212160    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-model-cache\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.212236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.212194    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-home\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.212289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.212270    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psl5d\" (UniqueName: \"kubernetes.io/projected/6c5a7abc-d740-431c-b70d-d0e7190f8833-kube-api-access-psl5d\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.262158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.262125    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"distinct-w2-e440bc-kserve-self-signed-certs\""
Nov 03 13:21:41.316458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.315412    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-tmp-dir\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.316458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.315500    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-model-cache\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.316458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.315547    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-home\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.316458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.315582    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psl5d\" (UniqueName: \"kubernetes.io/projected/6c5a7abc-d740-431c-b70d-d0e7190f8833-kube-api-access-psl5d\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.316458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.315691    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-dshm\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.316458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.315778    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5a7abc-d740-431c-b70d-d0e7190f8833-tls-certs\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.318880 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.318235    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-tmp-dir\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.347301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.340354    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-dshm\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.347301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.341845    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psl5d\" (UniqueName: \"kubernetes.io/projected/6c5a7abc-d740-431c-b70d-d0e7190f8833-kube-api-access-psl5d\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.347301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.342254    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-model-cache\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.347301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.342481    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-home\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.353185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.353146    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:41.360440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.360378    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5a7abc-d740-431c-b70d-d0e7190f8833-tls-certs\") pod \"distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.484416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.484378    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:41.732595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.732563    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:41.777265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.777178    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"]
Nov 03 13:21:41.777663 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:41.777632    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c5a7abc_d740_431c_b70d_d0e7190f8833.slice/crio-745e2ed4e083f44acb004d80f457fc4985a25a6d36c8e40be39016316c186760 WatchSource:0}: Error finding container 745e2ed4e083f44acb004d80f457fc4985a25a6d36c8e40be39016316c186760: Status 404 returned error can't find the container with id 745e2ed4e083f44acb004d80f457fc4985a25a6d36c8e40be39016316c186760
Nov 03 13:21:41.874298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:41.874265    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:42.037678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:42.037599    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:42.283956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:42.283922    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:42.459227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:42.459177    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:42.616365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:42.616332    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:42.790664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:42.790553    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" event={"ID":"6c5a7abc-d740-431c-b70d-d0e7190f8833","Type":"ContainerStarted","Data":"0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e"}
Nov 03 13:21:42.790664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:42.790611    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" event={"ID":"6c5a7abc-d740-431c-b70d-d0e7190f8833","Type":"ContainerStarted","Data":"745e2ed4e083f44acb004d80f457fc4985a25a6d36c8e40be39016316c186760"}
Nov 03 13:21:42.813902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:42.813854    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" podStartSLOduration=1.813840127 podStartE2EDuration="1.813840127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:42.812597326 +0000 UTC m=+743.890362352" watchObservedRunningTime="2026-11-03 13:21:42.813840127 +0000 UTC m=+743.891605154"
Nov 03 13:21:42.910801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:42.910768    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:43.348401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:43.348366    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:43.485399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:43.485360    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:43.496850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:43.496822    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:43.865860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:43.865825    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:44.214827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.214782    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"]
Nov 03 13:21:44.215592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.215548    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:44.219646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.219630    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.225849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.225827    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-29dd648f7d07\""
Nov 03 13:21:44.225849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.225840    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"kube-root-ca.crt\""
Nov 03 13:21:44.226072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.225873    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-e2e--cd1da41a-dockercfg-txc88\""
Nov 03 13:21:44.226072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.225917    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"routing-overlay\""
Nov 03 13:21:44.226072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.225960    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c192b\""
Nov 03 13:21:44.226072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.225970    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"openshift-service-ca.crt\""
Nov 03 13:21:44.227818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.227796    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"]
Nov 03 13:21:44.371236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.370774    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.373068 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.373034    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.373252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.373081    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.373252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.373112    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9szf\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-kube-api-access-r9szf\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.474520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.474440    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.474756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.474554    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.474756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.474584    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.474756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.474611    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9szf\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-kube-api-access-r9szf\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.475473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.475448    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.475473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.475460    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.477647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.477621    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.482829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.482805    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9szf\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-kube-api-access-r9szf\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.537398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.537103    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:44.705988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.705959    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:44.833917 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:44.833876    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884e4c72_728e_41be_88eb_5bfc9952a3b0.slice/crio-3f960f8f41b6285ef69c7afc8e3a68f808126c872931130e6b9ea2ee532a4a52 WatchSource:0}: Error finding container 3f960f8f41b6285ef69c7afc8e3a68f808126c872931130e6b9ea2ee532a4a52: Status 404 returned error can't find the container with id 3f960f8f41b6285ef69c7afc8e3a68f808126c872931130e6b9ea2ee532a4a52
Nov 03 13:21:44.834878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:44.834851    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"]
Nov 03 13:21:45.322594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:45.322555    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:45.696885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:45.696796    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:45.826165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:45.826116    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd" event={"ID":"884e4c72-728e-41be-88eb-5bfc9952a3b0","Type":"ContainerStarted","Data":"9a6d29bf000c5bea2f1514f9933634b7a0d4261f68649272f167185b8524c23b"}
Nov 03 13:21:45.826372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:45.826229    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd" event={"ID":"884e4c72-728e-41be-88eb-5bfc9952a3b0","Type":"ContainerStarted","Data":"3f960f8f41b6285ef69c7afc8e3a68f808126c872931130e6b9ea2ee532a4a52"}
Nov 03 13:21:45.826416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:45.826374    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:45.847633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:45.847584    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd" podStartSLOduration=1.847571042 podStartE2EDuration="1.847571042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:45.84355752 +0000 UTC m=+746.921322546" watchObservedRunningTime="2026-11-03 13:21:45.847571042 +0000 UTC m=+746.925336124"
Nov 03 13:21:45.912095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:45.912064    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:46.262817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:46.262784    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:46.431687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:46.431654    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:46.726963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:46.726926    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:47.394531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:47.394497    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:48.001675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:48.001641    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:48.601054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:48.601017    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:49.203622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:49.203589    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:50.035761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:50.035727    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:50.398446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:50.398361    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:50.565410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:50.565362    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:51.488235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:51.485016    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:51.540244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:51.538892    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:52.366236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:52.354092    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:52.424286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:52.422578    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:52.684431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:52.684323    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:52.856235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:52.833538    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:52.883643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:52.856541    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:21:52.972143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:52.972088    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 03 13:21:52.972525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:52.972498    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 13:21:53.044273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.044246    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:21:53.049035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.048967    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:53.064317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.064265    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"]
Nov 03 13:21:53.073153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.073125    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:21:53.082936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.082899    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a2ccd8\""
Nov 03 13:21:53.083089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.083064    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a2ccd8-dockercfg-rvh8g\""
Nov 03 13:21:53.093069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.093038    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"]
Nov 03 13:21:53.096147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.096127    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"]
Nov 03 13:21:53.113593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.113562    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:21:53.138175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.137694    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"]
Nov 03 13:21:53.178559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.178529    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttn69\" (UniqueName: \"kubernetes.io/projected/284d1e11-52e5-4c88-925a-35d22f1239d9-kube-api-access-ttn69\") pod \"payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9\" (UID: \"284d1e11-52e5-4c88-925a-35d22f1239d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:21:53.178738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.178600    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qqjj\" (UniqueName: \"kubernetes.io/projected/263fca38-ec96-41fe-9a14-5346ffd41192-kube-api-access-5qqjj\") pod \"payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz\" (UID: \"263fca38-ec96-41fe-9a14-5346ffd41192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:21:53.178860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.178840    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263fca38-ec96-41fe-9a14-5346ffd41192-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz\" (UID: \"263fca38-ec96-41fe-9a14-5346ffd41192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:21:53.178906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.178877    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284d1e11-52e5-4c88-925a-35d22f1239d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9\" (UID: \"284d1e11-52e5-4c88-925a-35d22f1239d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:21:53.211937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.211909    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:53.280290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.280055    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263fca38-ec96-41fe-9a14-5346ffd41192-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz\" (UID: \"263fca38-ec96-41fe-9a14-5346ffd41192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:21:53.280290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.280148    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284d1e11-52e5-4c88-925a-35d22f1239d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9\" (UID: \"284d1e11-52e5-4c88-925a-35d22f1239d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:21:53.280290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.280275    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttn69\" (UniqueName: \"kubernetes.io/projected/284d1e11-52e5-4c88-925a-35d22f1239d9-kube-api-access-ttn69\") pod \"payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9\" (UID: \"284d1e11-52e5-4c88-925a-35d22f1239d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:21:53.280579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.280365    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qqjj\" (UniqueName: \"kubernetes.io/projected/263fca38-ec96-41fe-9a14-5346ffd41192-kube-api-access-5qqjj\") pod \"payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz\" (UID: \"263fca38-ec96-41fe-9a14-5346ffd41192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:21:53.288235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.281563    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263fca38-ec96-41fe-9a14-5346ffd41192-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz\" (UID: \"263fca38-ec96-41fe-9a14-5346ffd41192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:21:53.288235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.282522    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284d1e11-52e5-4c88-925a-35d22f1239d9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9\" (UID: \"284d1e11-52e5-4c88-925a-35d22f1239d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:21:53.291429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.291078    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qqjj\" (UniqueName: \"kubernetes.io/projected/263fca38-ec96-41fe-9a14-5346ffd41192-kube-api-access-5qqjj\") pod \"payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz\" (UID: \"263fca38-ec96-41fe-9a14-5346ffd41192\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:21:53.293131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.293095    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttn69\" (UniqueName: \"kubernetes.io/projected/284d1e11-52e5-4c88-925a-35d22f1239d9-kube-api-access-ttn69\") pod \"payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9\" (UID: \"284d1e11-52e5-4c88-925a-35d22f1239d9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:21:53.414590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.414560    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:21:53.439542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.439377    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:21:53.716074 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:53.716032    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263fca38_ec96_41fe_9a14_5346ffd41192.slice/crio-43e431921ab0116a9328374eba3aad2a68d3dc53548ed0f7c85a25654db113e7 WatchSource:0}: Error finding container 43e431921ab0116a9328374eba3aad2a68d3dc53548ed0f7c85a25654db113e7: Status 404 returned error can't find the container with id 43e431921ab0116a9328374eba3aad2a68d3dc53548ed0f7c85a25654db113e7
Nov 03 13:21:53.716988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.716962    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"]
Nov 03 13:21:53.720579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.720554    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:53.785144 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:53.785112    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284d1e11_52e5_4c88_925a_35d22f1239d9.slice/crio-bbbfb3982ab015b2a05b5a8a04324214ef524028cccc1a0dde337f8935512a33 WatchSource:0}: Error finding container bbbfb3982ab015b2a05b5a8a04324214ef524028cccc1a0dde337f8935512a33: Status 404 returned error can't find the container with id bbbfb3982ab015b2a05b5a8a04324214ef524028cccc1a0dde337f8935512a33
Nov 03 13:21:53.787197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:53.787170    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"]
Nov 03 13:21:54.056144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.056049    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9" event={"ID":"284d1e11-52e5-4c88-925a-35d22f1239d9","Type":"ContainerStarted","Data":"f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14"}
Nov 03 13:21:54.056144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.056107    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9" event={"ID":"284d1e11-52e5-4c88-925a-35d22f1239d9","Type":"ContainerStarted","Data":"bbbfb3982ab015b2a05b5a8a04324214ef524028cccc1a0dde337f8935512a33"}
Nov 03 13:21:54.056389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.056247    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:21:54.057817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.057789    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz" event={"ID":"263fca38-ec96-41fe-9a14-5346ffd41192","Type":"ContainerStarted","Data":"336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80"}
Nov 03 13:21:54.057937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.057834    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:21:54.057937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.057851    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz" event={"ID":"263fca38-ec96-41fe-9a14-5346ffd41192","Type":"ContainerStarted","Data":"43e431921ab0116a9328374eba3aad2a68d3dc53548ed0f7c85a25654db113e7"}
Nov 03 13:21:54.074938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.074896    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9" podStartSLOduration=1.07488116 podStartE2EDuration="1.07488116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:54.073446787 +0000 UTC m=+755.151211812" watchObservedRunningTime="2026-11-03 13:21:54.07488116 +0000 UTC m=+755.152646186"
Nov 03 13:21:54.096270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.096229    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz" podStartSLOduration=1.096204215 podStartE2EDuration="1.096204215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:54.093271798 +0000 UTC m=+755.171036823" watchObservedRunningTime="2026-11-03 13:21:54.096204215 +0000 UTC m=+755.173969241"
Nov 03 13:21:54.214824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.214175    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"]
Nov 03 13:21:54.218615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.218590    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.227521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.226726    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7c192b-openshift-default-dockercfg-rx6fs\""
Nov 03 13:21:54.231558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.231525    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"]
Nov 03 13:21:54.296501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.296451    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.296721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.296507    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.296721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.296536    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.296840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.296734    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.296840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.296787    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.296946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.296921    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.297004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.296952    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.297004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.296973    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9hx\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-kube-api-access-db9hx\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.297127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.297034    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.397942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.397849    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.397942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.397887    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.397942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.397913    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.397988    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398031    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398122    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398169    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398198    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9hx\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-kube-api-access-db9hx\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398327    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398395    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398451    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398607    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398643    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.398957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.398940    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.400468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.400447    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.400706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.400690    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.406141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.406117    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.406236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.406181    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9hx\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-kube-api-access-db9hx\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.489785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.489733    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:54.535704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.535668    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:54.731631 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:21:54.731596    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:21:54.838021 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:54.837986    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15be20d9_c7e8_4696_a9cb_3395faffbd9d.slice/crio-e113b91f41902a267f4b676cc41e1e23a8b3eb30245ad78dfebc241ada948943 WatchSource:0}: Error finding container e113b91f41902a267f4b676cc41e1e23a8b3eb30245ad78dfebc241ada948943: Status 404 returned error can't find the container with id e113b91f41902a267f4b676cc41e1e23a8b3eb30245ad78dfebc241ada948943
Nov 03 13:21:54.841700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.841651    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:54.842238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.841811    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:54.842238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.841903    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:21:54.852347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:54.852321    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"]
Nov 03 13:21:55.067533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:55.067434    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" event={"ID":"15be20d9-c7e8-4696-a9cb-3395faffbd9d","Type":"ContainerStarted","Data":"50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135"}
Nov 03 13:21:55.067533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:55.067499    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" event={"ID":"15be20d9-c7e8-4696-a9cb-3395faffbd9d","Type":"ContainerStarted","Data":"e113b91f41902a267f4b676cc41e1e23a8b3eb30245ad78dfebc241ada948943"}
Nov 03 13:21:55.093437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:55.093381    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" podStartSLOduration=1.093367806 podStartE2EDuration="1.093367806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:55.08876777 +0000 UTC m=+756.166532805" watchObservedRunningTime="2026-11-03 13:21:55.093367806 +0000 UTC m=+756.171132832"
Nov 03 13:21:55.117669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:55.117633    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:55.545234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:55.544277    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:21:55.545234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:55.545136    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:21:55.545234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:55.545186    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 13:21:55.718350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:55.718321    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:56.507273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:56.507235    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:56.539235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:56.538614    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:21:56.539235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:56.538674    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 13:21:56.846628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:56.846528    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:21:57.313228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:57.313180    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:57.446304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:57.441657    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:57.545440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:57.537665    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:21:57.545440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:57.537726    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 13:21:57.643300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:57.641933    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:57.953413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:57.953383    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"]
Nov 03 13:21:57.958626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:57.958601    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:57.958826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:57.958804    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"]
Nov 03 13:21:57.963189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:57.962934    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"distinct-2-w2-e440bc-kserve-self-signed-certs\""
Nov 03 13:21:58.053415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.053387    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:58.083241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.083195    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-dshm\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.083416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.083284    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tmp-dir\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.083416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.083378    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tls-certs\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.083528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.083509    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-model-cache\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.083564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.083545    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhmw\" (UniqueName: \"kubernetes.io/projected/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-kube-api-access-6nhmw\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.083698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.083682    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-home\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.185434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.185398    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tls-certs\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.185644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.185473    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-model-cache\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.185644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.185507    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhmw\" (UniqueName: \"kubernetes.io/projected/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-kube-api-access-6nhmw\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.185644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.185611    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-home\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.185803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.185661    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-dshm\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.185803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.185683    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tmp-dir\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.185981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.185959    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-model-cache\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.186254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.186234    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-home\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.186329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.186293    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tmp-dir\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.187874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.187845    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:58.188938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.188914    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tls-certs\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.193938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.193909    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-dshm\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.198892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.198719    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhmw\" (UniqueName: \"kubernetes.io/projected/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-kube-api-access-6nhmw\") pod \"distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") " pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.280188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.279521    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:58.545171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.544629    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:21:58.545171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.544696    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 13:21:58.599223 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:21:58.599169    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5750fc35_ee3b_4539_a8e1_ad04af8d50b6.slice/crio-50d0067b9aa542d6b276f36c3ed47a94d457a236bdecc01453a4ab8465ed8e34 WatchSource:0}: Error finding container 50d0067b9aa542d6b276f36c3ed47a94d457a236bdecc01453a4ab8465ed8e34: Status 404 returned error can't find the container with id 50d0067b9aa542d6b276f36c3ed47a94d457a236bdecc01453a4ab8465ed8e34
Nov 03 13:21:58.599813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:58.599746    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"]
Nov 03 13:21:59.032313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:59.020191    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:59.103150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:59.103116    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" event={"ID":"5750fc35-ee3b-4539-a8e1-ad04af8d50b6","Type":"ContainerStarted","Data":"67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976"}
Nov 03 13:21:59.103150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:59.103161    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" event={"ID":"5750fc35-ee3b-4539-a8e1-ad04af8d50b6","Type":"ContainerStarted","Data":"50d0067b9aa542d6b276f36c3ed47a94d457a236bdecc01453a4ab8465ed8e34"}
Nov 03 13:21:59.281396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:59.281365    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:21:59.543236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:59.542611    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:21:59.543236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:59.542672    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 13:21:59.872183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:21:59.870534    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:00.540228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:00.536674    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:22:00.540228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:00.536734    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 13:22:00.540228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:00.539587    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:01.199636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:01.199578    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:01.542285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:01.542255    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:22:01.542585 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:01.542569    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:22:01.543301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:01.543283    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:22:01.569028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:01.568976    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" podStartSLOduration=4.56895983 podStartE2EDuration="4.56895983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:59.146424517 +0000 UTC m=+760.224189542" watchObservedRunningTime="2026-11-03 13:22:01.56895983 +0000 UTC m=+762.646724855"
Nov 03 13:22:02.108583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:02.108511    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:02.690158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:02.690124    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:03.384896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:03.384859    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:03.993965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:03.993934    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:04.848731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:04.848698    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:05.071333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:05.071303    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:22:05.071705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:05.071373    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:22:05.310714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:05.310654    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:05.894862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:05.894832    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:06.056917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:06.056886    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:22:06.608748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:06.608715    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:07.305715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:07.305682    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:08.286933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:08.285493    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:22:08.355013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:08.354975    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:08.818579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:08.818551    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:22:09.174765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:09.174682    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:09.232590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:09.232560    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:22:09.929443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:09.929410    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:10.603792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:10.603759    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:11.250613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:11.250580    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:11.934651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:11.934622    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:12.495820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:12.495789    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:13.181133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:13.181092    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:13.885063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:13.885035    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:14.465830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:14.465799    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:15.202190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.202154    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:15.537291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.537257    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"]
Nov 03 13:22:15.543039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.543010    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.547307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.547269    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:22:15.548682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.548656    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"]
Nov 03 13:22:15.549013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.548992    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:22:15.549123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.549008    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:22:15.549123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.549017    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:22:15.591462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.591430    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1641ad3-zmb7c\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.591633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.591492    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1641ad3-zmb7c\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.591633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.591611    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5lrc\" (UniqueName: \"kubernetes.io/projected/f159b56b-47a7-4352-ae6d-459b9402aa93-kube-api-access-g5lrc\") pod \"maas-api-revoke-sub-d1641ad3-zmb7c\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.693132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.693102    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1641ad3-zmb7c\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.693325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.693161    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1641ad3-zmb7c\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.693325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.693200    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5lrc\" (UniqueName: \"kubernetes.io/projected/f159b56b-47a7-4352-ae6d-459b9402aa93-kube-api-access-g5lrc\") pod \"maas-api-revoke-sub-d1641ad3-zmb7c\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.693821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.693803    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1641ad3-zmb7c\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.695405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.695388    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1641ad3-zmb7c\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.702644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.702623    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5lrc\" (UniqueName: \"kubernetes.io/projected/f159b56b-47a7-4352-ae6d-459b9402aa93-kube-api-access-g5lrc\") pod \"maas-api-revoke-sub-d1641ad3-zmb7c\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.716933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.716898    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:15.865485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.865411    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:15.916699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:15.916645    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:16.171599 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:16.171568    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf159b56b_47a7_4352_ae6d_459b9402aa93.slice/crio-82771b3f526f5fef8f72c5af19739fac6bbce002313d4244ee96202dc4c1102c WatchSource:0}: Error finding container 82771b3f526f5fef8f72c5af19739fac6bbce002313d4244ee96202dc4c1102c: Status 404 returned error can't find the container with id 82771b3f526f5fef8f72c5af19739fac6bbce002313d4244ee96202dc4c1102c
Nov 03 13:22:16.172168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:16.172144    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"]
Nov 03 13:22:16.275222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:16.275181    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c" event={"ID":"f159b56b-47a7-4352-ae6d-459b9402aa93","Type":"ContainerStarted","Data":"82771b3f526f5fef8f72c5af19739fac6bbce002313d4244ee96202dc4c1102c"}
Nov 03 13:22:16.505327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:16.505282    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:16.983299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:16.983266    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:17.155194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.155155    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:17.284661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.284624    5420 generic.go:356] "Generic (PLEG): container finished" podID="f159b56b-47a7-4352-ae6d-459b9402aa93" containerID="1b336f459506efd0ecca35018e684f02ef55531517ac06dc1990a59749c0eaed" exitCode=0
Nov 03 13:22:17.285125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.284713    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c" event={"ID":"f159b56b-47a7-4352-ae6d-459b9402aa93","Type":"ContainerDied","Data":"1b336f459506efd0ecca35018e684f02ef55531517ac06dc1990a59749c0eaed"}
Nov 03 13:22:17.482952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.482920    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"]
Nov 03 13:22:17.487432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.487413    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.500658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.500637    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b26c22-openshift-default-dockercfg-jmpmc\""
Nov 03 13:22:17.524490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.524455    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"]
Nov 03 13:22:17.618869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.618747    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:17.621932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.621834    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-data\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.622261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.622236    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-socket\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.622425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.622410    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-certs\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.622605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.622587    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-credential-socket\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.622780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.622766    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-istio-token\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.622889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.622875    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bdcc44-6914-4774-9b98-2f77caa51434-istiod-ca-cert\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.622993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.622978    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-envoy\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.623094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.623081    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxpd\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-kube-api-access-hsxpd\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.623258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.623242    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bdcc44-6914-4774-9b98-2f77caa51434-istio-podinfo\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.724793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.724759    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-data\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.724855    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-socket\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.724906    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-certs\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.724981    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-credential-socket\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.725066    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-istio-token\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.725096    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bdcc44-6914-4774-9b98-2f77caa51434-istiod-ca-cert\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.725132    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-envoy\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.725157    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxpd\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-kube-api-access-hsxpd\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.725254    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bdcc44-6914-4774-9b98-2f77caa51434-istio-podinfo\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.725321    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-data\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.725429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.725360    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-certs\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.726044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.726019    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-socket\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.726044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.726026    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bdcc44-6914-4774-9b98-2f77caa51434-istiod-ca-cert\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.727165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.726156    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-credential-socket\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.728002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.727977    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-envoy\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.728189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.728168    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bdcc44-6914-4774-9b98-2f77caa51434-istio-podinfo\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.737305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.737281    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxpd\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-kube-api-access-hsxpd\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.737643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.737622    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-istio-token\") pod \"e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") " pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:17.774842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.774806    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:17.801571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:17.801534    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:18.105268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.105236    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:18.112600 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:18.112570    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bdcc44_6914_4774_9b98_2f77caa51434.slice/crio-80734a1475a73f2cdb2b2a8973ceccd921dac8afeb0f9508888af7e25111aa50 WatchSource:0}: Error finding container 80734a1475a73f2cdb2b2a8973ceccd921dac8afeb0f9508888af7e25111aa50: Status 404 returned error can't find the container with id 80734a1475a73f2cdb2b2a8973ceccd921dac8afeb0f9508888af7e25111aa50
Nov 03 13:22:18.113272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.113248    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"]
Nov 03 13:22:18.115206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.115176    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:18.115283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.115258    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:18.115319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.115291    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:18.294272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.294206    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" event={"ID":"75bdcc44-6914-4774-9b98-2f77caa51434","Type":"ContainerStarted","Data":"7d27bffe5874fb86ddbf72c3926b9f3dec917c210d09394ecc88511098123ebb"}
Nov 03 13:22:18.294714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.294291    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" event={"ID":"75bdcc44-6914-4774-9b98-2f77caa51434","Type":"ContainerStarted","Data":"80734a1475a73f2cdb2b2a8973ceccd921dac8afeb0f9508888af7e25111aa50"}
Nov 03 13:22:18.317592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.317544    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" podStartSLOduration=1.317530067 podStartE2EDuration="1.317530067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:18.314140685 +0000 UTC m=+779.391905710" watchObservedRunningTime="2026-11-03 13:22:18.317530067 +0000 UTC m=+779.395295148"
Nov 03 13:22:18.445127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.445094    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:18.540112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.540084    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f159b56b-47a7-4352-ae6d-459b9402aa93-kube-api-access-g5lrc\" (UniqueName: \"kubernetes.io/projected/f159b56b-47a7-4352-ae6d-459b9402aa93-kube-api-access-g5lrc\") pod \"f159b56b-47a7-4352-ae6d-459b9402aa93\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") "
Nov 03 13:22:18.540307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.540181    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-openshift-ca\") pod \"f159b56b-47a7-4352-ae6d-459b9402aa93\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") "
Nov 03 13:22:18.540307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.540265    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-xks-ca\") pod \"f159b56b-47a7-4352-ae6d-459b9402aa93\" (UID: \"f159b56b-47a7-4352-ae6d-459b9402aa93\") "
Nov 03 13:22:18.540593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.540554    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-openshift-ca" pod "f159b56b-47a7-4352-ae6d-459b9402aa93" (UID: "f159b56b-47a7-4352-ae6d-459b9402aa93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:18.540864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.540845    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:18.542306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.542282    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-xks-ca" pod "f159b56b-47a7-4352-ae6d-459b9402aa93" (UID: "f159b56b-47a7-4352-ae6d-459b9402aa93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:18.542306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.542299    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f159b56b-47a7-4352-ae6d-459b9402aa93-kube-api-access-g5lrc" pod "f159b56b-47a7-4352-ae6d-459b9402aa93" (UID: "f159b56b-47a7-4352-ae6d-459b9402aa93"). InnerVolumeSpecName "kube-api-access-g5lrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:18.642107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.642071    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f159b56b-47a7-4352-ae6d-459b9402aa93-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:18.642107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.642104    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5lrc\" (UniqueName: \"kubernetes.io/projected/f159b56b-47a7-4352-ae6d-459b9402aa93-kube-api-access-g5lrc\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:18.802463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.802434    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:18.831966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:18.831931    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:19.302665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:19.302634    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"
Nov 03 13:22:19.302665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:19.302651    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c" event={"ID":"f159b56b-47a7-4352-ae6d-459b9402aa93","Type":"ContainerDied","Data":"82771b3f526f5fef8f72c5af19739fac6bbce002313d4244ee96202dc4c1102c"}
Nov 03 13:22:19.303067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:19.302729    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82771b3f526f5fef8f72c5af19739fac6bbce002313d4244ee96202dc4c1102c"
Nov 03 13:22:19.398817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:19.398730    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:19.807812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:19.807782    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:20.120167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.120085    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:20.310520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.310492    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:20.311407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.311388    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:20.593503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.593464    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:20.731883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.731851    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:20.891747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.891663    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:20.904010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.903976    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"]
Nov 03 13:22:20.904637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.904622    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f159b56b-47a7-4352-ae6d-459b9402aa93" containerName="revoke-keys"
Nov 03 13:22:20.904708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.904639    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f159b56b-47a7-4352-ae6d-459b9402aa93" containerName="revoke-keys"
Nov 03 13:22:20.904755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.904746    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f159b56b-47a7-4352-ae6d-459b9402aa93" containerName="revoke-keys"
Nov 03 13:22:20.910326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.909918    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:20.916170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.916140    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"]
Nov 03 13:22:20.916742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.916665    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:22:20.916742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.916697    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:22:20.917125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.917109    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:22:20.917276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.917256    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:22:20.979193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.979156    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9tcj\" (UniqueName: \"kubernetes.io/projected/63591942-3ae9-49bd-ae74-293ef518d990-kube-api-access-l9tcj\") pod \"maas-api-revoke-sub-4c28f80a-fwvlx\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:20.979193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.979194    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c28f80a-fwvlx\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:20.979445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:20.979238    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c28f80a-fwvlx\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:21.080231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.080180    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9tcj\" (UniqueName: \"kubernetes.io/projected/63591942-3ae9-49bd-ae74-293ef518d990-kube-api-access-l9tcj\") pod \"maas-api-revoke-sub-4c28f80a-fwvlx\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:21.080231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.080232    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c28f80a-fwvlx\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:21.080473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.080262    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c28f80a-fwvlx\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:21.081007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.080979    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c28f80a-fwvlx\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:21.082790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.082770    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c28f80a-fwvlx\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:21.088866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.088841    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9tcj\" (UniqueName: \"kubernetes.io/projected/63591942-3ae9-49bd-ae74-293ef518d990-kube-api-access-l9tcj\") pod \"maas-api-revoke-sub-4c28f80a-fwvlx\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:21.234329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.234296    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:21.494839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.494766    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:21.524505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.524475    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"]
Nov 03 13:22:21.527526 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:21.527497    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63591942_3ae9_49bd_ae74_293ef518d990.slice/crio-295ac8464c7c850f49eabe600fac0f2bf9c5ffd1b8dd3e155f2d807efc8ce0f2 WatchSource:0}: Error finding container 295ac8464c7c850f49eabe600fac0f2bf9c5ffd1b8dd3e155f2d807efc8ce0f2: Status 404 returned error can't find the container with id 295ac8464c7c850f49eabe600fac0f2bf9c5ffd1b8dd3e155f2d807efc8ce0f2
Nov 03 13:22:21.877656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.877625    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:21.950381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.950341    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"]
Nov 03 13:22:21.954716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.954694    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:21.963631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.963606    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"]
Nov 03 13:22:21.994115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.994081    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9mj\" (UniqueName: \"kubernetes.io/projected/03557d4b-676e-4fe8-88a1-5695df46ccc9-kube-api-access-lx9mj\") pod \"payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc\" (UID: \"03557d4b-676e-4fe8-88a1-5695df46ccc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:21.994384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:21.994347    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03557d4b-676e-4fe8-88a1-5695df46ccc9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc\" (UID: \"03557d4b-676e-4fe8-88a1-5695df46ccc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:22.016120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.016094    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b26c22-dockercfg-jf68r\""
Nov 03 13:22:22.016317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.016134    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b26c22\""
Nov 03 13:22:22.032925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.032891    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:22.096405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.096312    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03557d4b-676e-4fe8-88a1-5695df46ccc9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc\" (UID: \"03557d4b-676e-4fe8-88a1-5695df46ccc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:22.096575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.096513    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9mj\" (UniqueName: \"kubernetes.io/projected/03557d4b-676e-4fe8-88a1-5695df46ccc9-kube-api-access-lx9mj\") pod \"payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc\" (UID: \"03557d4b-676e-4fe8-88a1-5695df46ccc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:22.098202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.098170    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03557d4b-676e-4fe8-88a1-5695df46ccc9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc\" (UID: \"03557d4b-676e-4fe8-88a1-5695df46ccc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:22.107305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.107284    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9mj\" (UniqueName: \"kubernetes.io/projected/03557d4b-676e-4fe8-88a1-5695df46ccc9-kube-api-access-lx9mj\") pod \"payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc\" (UID: \"03557d4b-676e-4fe8-88a1-5695df46ccc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:22.270568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.270292    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:22.286319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.286268    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:22.347772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.347614    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx" event={"ID":"63591942-3ae9-49bd-ae74-293ef518d990","Type":"ContainerStarted","Data":"d8e1d79772c3af2fe51d8e0c59f4640a435351e83d40cea09493761b15242c7a"}
Nov 03 13:22:22.347772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.347675    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx" event={"ID":"63591942-3ae9-49bd-ae74-293ef518d990","Type":"ContainerStarted","Data":"295ac8464c7c850f49eabe600fac0f2bf9c5ffd1b8dd3e155f2d807efc8ce0f2"}
Nov 03 13:22:22.403015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.402945    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx" podStartSLOduration=2.40292525 podStartE2EDuration="2.40292525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:22.380226 +0000 UTC m=+783.457991024" watchObservedRunningTime="2026-11-03 13:22:22.40292525 +0000 UTC m=+783.480690276"
Nov 03 13:22:22.595743 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:22.595710    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03557d4b_676e_4fe8_88a1_5695df46ccc9.slice/crio-80ce2bd8c65081bcf61b8811504acf46b2822ca53fa5a5fce7af96e7a88fed61 WatchSource:0}: Error finding container 80ce2bd8c65081bcf61b8811504acf46b2822ca53fa5a5fce7af96e7a88fed61: Status 404 returned error can't find the container with id 80ce2bd8c65081bcf61b8811504acf46b2822ca53fa5a5fce7af96e7a88fed61
Nov 03 13:22:22.596175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.595947    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"]
Nov 03 13:22:22.904124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:22.904026    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:23.357809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:23.357778    5420 generic.go:356] "Generic (PLEG): container finished" podID="63591942-3ae9-49bd-ae74-293ef518d990" containerID="d8e1d79772c3af2fe51d8e0c59f4640a435351e83d40cea09493761b15242c7a" exitCode=0
Nov 03 13:22:23.357983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:23.357866    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx" event={"ID":"63591942-3ae9-49bd-ae74-293ef518d990","Type":"ContainerDied","Data":"d8e1d79772c3af2fe51d8e0c59f4640a435351e83d40cea09493761b15242c7a"}
Nov 03 13:22:23.359594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:23.359569    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc" event={"ID":"03557d4b-676e-4fe8-88a1-5695df46ccc9","Type":"ContainerStarted","Data":"195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d"}
Nov 03 13:22:23.359703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:23.359609    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc" event={"ID":"03557d4b-676e-4fe8-88a1-5695df46ccc9","Type":"ContainerStarted","Data":"80ce2bd8c65081bcf61b8811504acf46b2822ca53fa5a5fce7af96e7a88fed61"}
Nov 03 13:22:23.359703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:23.359645    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:23.496685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:23.496655    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:24.039691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.039639    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc" podStartSLOduration=3.039624053 podStartE2EDuration="3.039624053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:23.395005378 +0000 UTC m=+784.472770403" watchObservedRunningTime="2026-11-03 13:22:24.039624053 +0000 UTC m=+785.117389079"
Nov 03 13:22:24.040249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.040230    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:22:24.103933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.103897    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:24.517019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.516996    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:24.639279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.632295    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63591942-3ae9-49bd-ae74-293ef518d990-kube-api-access-l9tcj\" (UniqueName: \"kubernetes.io/projected/63591942-3ae9-49bd-ae74-293ef518d990-kube-api-access-l9tcj\") pod \"63591942-3ae9-49bd-ae74-293ef518d990\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") "
Nov 03 13:22:24.639279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.632876    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-xks-ca\") pod \"63591942-3ae9-49bd-ae74-293ef518d990\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") "
Nov 03 13:22:24.639279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.633529    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-openshift-ca\") pod \"63591942-3ae9-49bd-ae74-293ef518d990\" (UID: \"63591942-3ae9-49bd-ae74-293ef518d990\") "
Nov 03 13:22:24.639279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.633977    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-openshift-ca" pod "63591942-3ae9-49bd-ae74-293ef518d990" (UID: "63591942-3ae9-49bd-ae74-293ef518d990"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:24.639279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.634606    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:24.666230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.665987    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63591942-3ae9-49bd-ae74-293ef518d990-kube-api-access-l9tcj" pod "63591942-3ae9-49bd-ae74-293ef518d990" (UID: "63591942-3ae9-49bd-ae74-293ef518d990"). InnerVolumeSpecName "kube-api-access-l9tcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:24.685482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.685338    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-xks-ca" pod "63591942-3ae9-49bd-ae74-293ef518d990" (UID: "63591942-3ae9-49bd-ae74-293ef518d990"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:24.747780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.747740    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9tcj\" (UniqueName: \"kubernetes.io/projected/63591942-3ae9-49bd-ae74-293ef518d990-kube-api-access-l9tcj\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:24.747780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.747780    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63591942-3ae9-49bd-ae74-293ef518d990-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:24.770741 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:22:24.770701    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:22:24.785588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:24.785550    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:25.512896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:25.487469    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"
Nov 03 13:22:25.512896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:25.502696    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx" event={"ID":"63591942-3ae9-49bd-ae74-293ef518d990","Type":"ContainerDied","Data":"295ac8464c7c850f49eabe600fac0f2bf9c5ffd1b8dd3e155f2d807efc8ce0f2"}
Nov 03 13:22:25.512896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:25.502821    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="295ac8464c7c850f49eabe600fac0f2bf9c5ffd1b8dd3e155f2d807efc8ce0f2"
Nov 03 13:22:26.389238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:26.385329    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:26.412397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:26.412351    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:26.691287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:26.680316    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:26.827235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:26.824312    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 13:22:26.844231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:26.838470    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:26.849246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:26.848281    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:27.327237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:27.320143    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:27.446256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:27.444290    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:27.446256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:27.444360    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:27.516244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:27.516193    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:27.708355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:27.708318    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:28.107267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:28.107131    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:28.715245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:28.715193    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:29.313162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.313107    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:29.668478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.668408    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"]
Nov 03 13:22:29.668987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.668974    5420 cpu_manager.go:414] "containerMap: removing container" podUID="63591942-3ae9-49bd-ae74-293ef518d990" containerName="revoke-keys"
Nov 03 13:22:29.669029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.668988    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63591942-3ae9-49bd-ae74-293ef518d990" containerName="revoke-keys"
Nov 03 13:22:29.669117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.669108    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="63591942-3ae9-49bd-ae74-293ef518d990" containerName="revoke-keys"
Nov 03 13:22:29.674003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.673973    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.679777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.679727    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"]
Nov 03 13:22:29.680135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.680114    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:22:29.680135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.680114    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:22:29.680356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.680151    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:22:29.680356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.680152    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:22:29.830187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.830153    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrmn\" (UniqueName: \"kubernetes.io/projected/49920094-748c-490a-a7dc-ded3101add86-kube-api-access-htrmn\") pod \"maas-api-revoke-sub-fb77f064-vrw8l\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.830645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.830243    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49920094-748c-490a-a7dc-ded3101add86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb77f064-vrw8l\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.830645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.830343    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49920094-748c-490a-a7dc-ded3101add86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb77f064-vrw8l\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.931921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.931821    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49920094-748c-490a-a7dc-ded3101add86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb77f064-vrw8l\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.931921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.931891    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49920094-748c-490a-a7dc-ded3101add86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb77f064-vrw8l\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.932189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.932144    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrmn\" (UniqueName: \"kubernetes.io/projected/49920094-748c-490a-a7dc-ded3101add86-kube-api-access-htrmn\") pod \"maas-api-revoke-sub-fb77f064-vrw8l\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.932688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.932668    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49920094-748c-490a-a7dc-ded3101add86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb77f064-vrw8l\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.938325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.938294    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:29.940828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.940805    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49920094-748c-490a-a7dc-ded3101add86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb77f064-vrw8l\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.947035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.947008    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrmn\" (UniqueName: \"kubernetes.io/projected/49920094-748c-490a-a7dc-ded3101add86-kube-api-access-htrmn\") pod \"maas-api-revoke-sub-fb77f064-vrw8l\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:29.990794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:29.990764    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:30.273369 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:30.273339    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49920094_748c_490a_a7dc_ded3101add86.slice/crio-8262e9e90a55060c56a4083dc871cf482376697948f351a37f88a4551861de25 WatchSource:0}: Error finding container 8262e9e90a55060c56a4083dc871cf482376697948f351a37f88a4551861de25: Status 404 returned error can't find the container with id 8262e9e90a55060c56a4083dc871cf482376697948f351a37f88a4551861de25
Nov 03 13:22:30.274774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:30.274744    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"]
Nov 03 13:22:30.506173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:30.506123    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:30.603374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:30.603338    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l" event={"ID":"49920094-748c-490a-a7dc-ded3101add86","Type":"ContainerStarted","Data":"f13e199d1fc85065f7fcc90ebb021e2e8522f675ce31a78b7c346c2c9f5094d9"}
Nov 03 13:22:30.603584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:30.603402    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l" event={"ID":"49920094-748c-490a-a7dc-ded3101add86","Type":"ContainerStarted","Data":"8262e9e90a55060c56a4083dc871cf482376697948f351a37f88a4551861de25"}
Nov 03 13:22:30.625735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:30.625685    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l" podStartSLOduration=1.625671225 podStartE2EDuration="1.625671225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:30.623261945 +0000 UTC m=+791.701026972" watchObservedRunningTime="2026-11-03 13:22:30.625671225 +0000 UTC m=+791.703436253"
Nov 03 13:22:31.227118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:31.227075    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:31.560430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:31.560400    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:31.623801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:31.623767    5420 generic.go:356] "Generic (PLEG): container finished" podID="49920094-748c-490a-a7dc-ded3101add86" containerID="f13e199d1fc85065f7fcc90ebb021e2e8522f675ce31a78b7c346c2c9f5094d9" exitCode=0
Nov 03 13:22:31.623999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:31.623908    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l" event={"ID":"49920094-748c-490a-a7dc-ded3101add86","Type":"ContainerDied","Data":"f13e199d1fc85065f7fcc90ebb021e2e8522f675ce31a78b7c346c2c9f5094d9"}
Nov 03 13:22:31.967795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:31.967765    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"]
Nov 03 13:22:31.972269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:31.972250    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:31.978763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:31.978737    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"]
Nov 03 13:22:32.064026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.063989    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-320dd917-p678n\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.064187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.064162    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-320dd917-p678n\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.064296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.064282    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzpjh\" (UniqueName: \"kubernetes.io/projected/31ee43d5-98d0-4510-8558-848e406f9f90-kube-api-access-kzpjh\") pod \"maas-api-revoke-sub-320dd917-p678n\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.096860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.096818    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:32.165638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.165603    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-320dd917-p678n\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.165785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.165685    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzpjh\" (UniqueName: \"kubernetes.io/projected/31ee43d5-98d0-4510-8558-848e406f9f90-kube-api-access-kzpjh\") pod \"maas-api-revoke-sub-320dd917-p678n\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.165870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.165853    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-320dd917-p678n\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.166391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.166369    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-320dd917-p678n\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.168149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.168131    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-320dd917-p678n\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.173532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.173509    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzpjh\" (UniqueName: \"kubernetes.io/projected/31ee43d5-98d0-4510-8558-848e406f9f90-kube-api-access-kzpjh\") pod \"maas-api-revoke-sub-320dd917-p678n\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.290205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.290126    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:32.577864 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:32.577834    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ee43d5_98d0_4510_8558_848e406f9f90.slice/crio-f564defeeb7e0642abe3da7634809e45fe7c491b07901f890364ab943ad9201b WatchSource:0}: Error finding container f564defeeb7e0642abe3da7634809e45fe7c491b07901f890364ab943ad9201b: Status 404 returned error can't find the container with id f564defeeb7e0642abe3da7634809e45fe7c491b07901f890364ab943ad9201b
Nov 03 13:22:32.578266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.578235    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"]
Nov 03 13:22:32.633366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.633337    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n" event={"ID":"31ee43d5-98d0-4510-8558-848e406f9f90","Type":"ContainerStarted","Data":"f564defeeb7e0642abe3da7634809e45fe7c491b07901f890364ab943ad9201b"}
Nov 03 13:22:32.731019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.730984    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:32.791201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.791169    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:32.874529 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.874448    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49920094-748c-490a-a7dc-ded3101add86-kube-api-access-htrmn\" (UniqueName: \"kubernetes.io/projected/49920094-748c-490a-a7dc-ded3101add86-kube-api-access-htrmn\") pod \"49920094-748c-490a-a7dc-ded3101add86\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") "
Nov 03 13:22:32.874704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.874529    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49920094-748c-490a-a7dc-ded3101add86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49920094-748c-490a-a7dc-ded3101add86-maas-api-openshift-ca\") pod \"49920094-748c-490a-a7dc-ded3101add86\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") "
Nov 03 13:22:32.874704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.874661    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49920094-748c-490a-a7dc-ded3101add86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49920094-748c-490a-a7dc-ded3101add86-maas-api-xks-ca\") pod \"49920094-748c-490a-a7dc-ded3101add86\" (UID: \"49920094-748c-490a-a7dc-ded3101add86\") "
Nov 03 13:22:32.874931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.874907    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49920094-748c-490a-a7dc-ded3101add86-maas-api-openshift-ca" pod "49920094-748c-490a-a7dc-ded3101add86" (UID: "49920094-748c-490a-a7dc-ded3101add86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:32.875207 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.875190    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49920094-748c-490a-a7dc-ded3101add86-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:32.876608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.876588    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49920094-748c-490a-a7dc-ded3101add86-maas-api-xks-ca" pod "49920094-748c-490a-a7dc-ded3101add86" (UID: "49920094-748c-490a-a7dc-ded3101add86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:32.876712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.876693    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49920094-748c-490a-a7dc-ded3101add86-kube-api-access-htrmn" pod "49920094-748c-490a-a7dc-ded3101add86" (UID: "49920094-748c-490a-a7dc-ded3101add86"). InnerVolumeSpecName "kube-api-access-htrmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:32.976366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.976339    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrmn\" (UniqueName: \"kubernetes.io/projected/49920094-748c-490a-a7dc-ded3101add86-kube-api-access-htrmn\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:32.976481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:32.976386    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49920094-748c-490a-a7dc-ded3101add86-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:33.379491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:33.379460    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:33.642391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:33.642358    5420 generic.go:356] "Generic (PLEG): container finished" podID="31ee43d5-98d0-4510-8558-848e406f9f90" containerID="d8792e2ac3bb09737f20c018259cd3582e893b30f5ba5abbf3568ad9386f0ac9" exitCode=0
Nov 03 13:22:33.642566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:33.642442    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n" event={"ID":"31ee43d5-98d0-4510-8558-848e406f9f90","Type":"ContainerDied","Data":"d8792e2ac3bb09737f20c018259cd3582e893b30f5ba5abbf3568ad9386f0ac9"}
Nov 03 13:22:33.644072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:33.644054    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"
Nov 03 13:22:33.644202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:33.644084    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l" event={"ID":"49920094-748c-490a-a7dc-ded3101add86","Type":"ContainerDied","Data":"8262e9e90a55060c56a4083dc871cf482376697948f351a37f88a4551861de25"}
Nov 03 13:22:33.644202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:33.644126    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8262e9e90a55060c56a4083dc871cf482376697948f351a37f88a4551861de25"
Nov 03 13:22:33.983204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:33.983169    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:34.371049 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:34.370972    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:22:34.597204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:34.597165    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:34.825679 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:34.825657    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:35.004280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.004238    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-openshift-ca\") pod \"31ee43d5-98d0-4510-8558-848e406f9f90\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") "
Nov 03 13:22:35.004439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.004364    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ee43d5-98d0-4510-8558-848e406f9f90-kube-api-access-kzpjh\" (UniqueName: \"kubernetes.io/projected/31ee43d5-98d0-4510-8558-848e406f9f90-kube-api-access-kzpjh\") pod \"31ee43d5-98d0-4510-8558-848e406f9f90\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") "
Nov 03 13:22:35.004515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.004503    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-xks-ca\") pod \"31ee43d5-98d0-4510-8558-848e406f9f90\" (UID: \"31ee43d5-98d0-4510-8558-848e406f9f90\") "
Nov 03 13:22:35.004620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.004594    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-openshift-ca" pod "31ee43d5-98d0-4510-8558-848e406f9f90" (UID: "31ee43d5-98d0-4510-8558-848e406f9f90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:35.004928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.004909    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:35.006493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.006469    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-xks-ca" pod "31ee43d5-98d0-4510-8558-848e406f9f90" (UID: "31ee43d5-98d0-4510-8558-848e406f9f90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:35.006608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.006494    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ee43d5-98d0-4510-8558-848e406f9f90-kube-api-access-kzpjh" pod "31ee43d5-98d0-4510-8558-848e406f9f90" (UID: "31ee43d5-98d0-4510-8558-848e406f9f90"). InnerVolumeSpecName "kube-api-access-kzpjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:35.106089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.106061    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ee43d5-98d0-4510-8558-848e406f9f90-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:35.106089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.106084    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzpjh\" (UniqueName: \"kubernetes.io/projected/31ee43d5-98d0-4510-8558-848e406f9f90-kube-api-access-kzpjh\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:35.288144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.288046    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:35.606534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.606441    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:35.674869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.674838    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n" event={"ID":"31ee43d5-98d0-4510-8558-848e406f9f90","Type":"ContainerDied","Data":"f564defeeb7e0642abe3da7634809e45fe7c491b07901f890364ab943ad9201b"}
Nov 03 13:22:35.675038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.674933    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f564defeeb7e0642abe3da7634809e45fe7c491b07901f890364ab943ad9201b"
Nov 03 13:22:35.675038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.674956    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"
Nov 03 13:22:35.762232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.760139    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:35.960824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.960783    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"]
Nov 03 13:22:35.961669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.961647    5420 cpu_manager.go:414] "containerMap: removing container" podUID="49920094-748c-490a-a7dc-ded3101add86" containerName="revoke-keys"
Nov 03 13:22:35.961669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.961668    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49920094-748c-490a-a7dc-ded3101add86" containerName="revoke-keys"
Nov 03 13:22:35.961877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.961720    5420 cpu_manager.go:414] "containerMap: removing container" podUID="31ee43d5-98d0-4510-8558-848e406f9f90" containerName="revoke-keys"
Nov 03 13:22:35.961877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.961729    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ee43d5-98d0-4510-8558-848e406f9f90" containerName="revoke-keys"
Nov 03 13:22:35.961980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.961876    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="49920094-748c-490a-a7dc-ded3101add86" containerName="revoke-keys"
Nov 03 13:22:35.961980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.961896    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ee43d5-98d0-4510-8558-848e406f9f90" containerName="revoke-keys"
Nov 03 13:22:35.966419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.966376    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:35.969383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.969170    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"]
Nov 03 13:22:35.970733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.970651    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:22:35.971722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.971324    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:22:35.971722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.971422    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:22:35.971722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:35.971466    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:22:36.024670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.024640    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhtv6\" (UniqueName: \"kubernetes.io/projected/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-kube-api-access-zhtv6\") pod \"maas-api-revoke-sub-89922625-9qrgl\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.024827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.024770    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89922625-9qrgl\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.024872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.024854    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89922625-9qrgl\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.125766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.125722    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhtv6\" (UniqueName: \"kubernetes.io/projected/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-kube-api-access-zhtv6\") pod \"maas-api-revoke-sub-89922625-9qrgl\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.125954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.125842    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89922625-9qrgl\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.125954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.125908    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89922625-9qrgl\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.126663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.126642    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89922625-9qrgl\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.128186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.128166    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89922625-9qrgl\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.133771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.133751    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhtv6\" (UniqueName: \"kubernetes.io/projected/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-kube-api-access-zhtv6\") pod \"maas-api-revoke-sub-89922625-9qrgl\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.260669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.260579    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:36.278220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.278185    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"]
Nov 03 13:22:36.283843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.283806    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.284764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.284734    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"]
Nov 03 13:22:36.286393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.286369    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:36.287982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.287960    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-f5ffe280c5d9\""
Nov 03 13:22:36.331133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.331090    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.331340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.331226    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.331340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.331286    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.331476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.331460    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdgk\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-kube-api-access-vpdgk\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.432871 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.432840    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.433030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.432929    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.433030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.432972    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.433149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.433139    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdgk\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-kube-api-access-vpdgk\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.433697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.433656    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.433913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.433891    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.435860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.435834    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.449833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.449802    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdgk\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-kube-api-access-vpdgk\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.597878 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:36.597845    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9ab0f3_3e08_4d06_b4ac_ffc86671db84.slice/crio-a4d120d2568bfe4d2f75cb03a86c647dc71057bc425168ebe4faab10ca339c3c WatchSource:0}: Error finding container a4d120d2568bfe4d2f75cb03a86c647dc71057bc425168ebe4faab10ca339c3c: Status 404 returned error can't find the container with id a4d120d2568bfe4d2f75cb03a86c647dc71057bc425168ebe4faab10ca339c3c
Nov 03 13:22:36.598182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.598159    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"]
Nov 03 13:22:36.610547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.610522    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:36.692702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.692672    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl" event={"ID":"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84","Type":"ContainerStarted","Data":"a4d120d2568bfe4d2f75cb03a86c647dc71057bc425168ebe4faab10ca339c3c"}
Nov 03 13:22:36.901839 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:36.901792    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3080298_1815_4006_ae0e_fcd845f65ee7.slice/crio-b1bfbeb458c172369d0c3974d0b727360ee29bc275f7d04b6fbd862b92f6887b WatchSource:0}: Error finding container b1bfbeb458c172369d0c3974d0b727360ee29bc275f7d04b6fbd862b92f6887b: Status 404 returned error can't find the container with id b1bfbeb458c172369d0c3974d0b727360ee29bc275f7d04b6fbd862b92f6887b
Nov 03 13:22:36.905610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.902919    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"]
Nov 03 13:22:36.906793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:36.906765    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:37.530718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:37.530685    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:37.702368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:37.702260    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng" event={"ID":"f3080298-1815-4006-ae0e-fcd845f65ee7","Type":"ContainerStarted","Data":"946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885"}
Nov 03 13:22:37.702368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:37.702335    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:37.702368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:37.702363    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng" event={"ID":"f3080298-1815-4006-ae0e-fcd845f65ee7","Type":"ContainerStarted","Data":"b1bfbeb458c172369d0c3974d0b727360ee29bc275f7d04b6fbd862b92f6887b"}
Nov 03 13:22:37.703936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:37.703914    5420 generic.go:356] "Generic (PLEG): container finished" podID="3d9ab0f3-3e08-4d06-b4ac-ffc86671db84" containerID="ab63aceefb0345c11b6a247fec71d7033d9c9f8597d86f97e498e96ca393d345" exitCode=0
Nov 03 13:22:37.704054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:37.703988    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl" event={"ID":"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84","Type":"ContainerDied","Data":"ab63aceefb0345c11b6a247fec71d7033d9c9f8597d86f97e498e96ca393d345"}
Nov 03 13:22:37.723497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:37.723455    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng" podStartSLOduration=1.723440438 podStartE2EDuration="1.723440438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:37.721951025 +0000 UTC m=+798.799716050" watchObservedRunningTime="2026-11-03 13:22:37.723440438 +0000 UTC m=+798.801205461"
Nov 03 13:22:38.214174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.214127    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:38.803595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.803554    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:38.860566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.860537    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:38.868776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.868752    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-kube-api-access-zhtv6\" (UniqueName: \"kubernetes.io/projected/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-kube-api-access-zhtv6\") pod \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") "
Nov 03 13:22:38.868923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.868888    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-openshift-ca\") pod \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") "
Nov 03 13:22:38.868996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.868983    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-xks-ca\") pod \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\" (UID: \"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84\") "
Nov 03 13:22:38.869252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.869196    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-openshift-ca" pod "3d9ab0f3-3e08-4d06-b4ac-ffc86671db84" (UID: "3d9ab0f3-3e08-4d06-b4ac-ffc86671db84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:38.869497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.869480    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:38.871394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.871375    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-xks-ca" pod "3d9ab0f3-3e08-4d06-b4ac-ffc86671db84" (UID: "3d9ab0f3-3e08-4d06-b4ac-ffc86671db84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:38.871468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.871417    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-kube-api-access-zhtv6" pod "3d9ab0f3-3e08-4d06-b4ac-ffc86671db84" (UID: "3d9ab0f3-3e08-4d06-b4ac-ffc86671db84"). InnerVolumeSpecName "kube-api-access-zhtv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:38.979296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.977394    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhtv6\" (UniqueName: \"kubernetes.io/projected/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-kube-api-access-zhtv6\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:38.979296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:38.977439    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:39.303394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.303318    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"]
Nov 03 13:22:39.304335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.304314    5420 cpu_manager.go:414] "containerMap: removing container" podUID="3d9ab0f3-3e08-4d06-b4ac-ffc86671db84" containerName="revoke-keys"
Nov 03 13:22:39.304397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.304340    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9ab0f3-3e08-4d06-b4ac-ffc86671db84" containerName="revoke-keys"
Nov 03 13:22:39.304555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.304541    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9ab0f3-3e08-4d06-b4ac-ffc86671db84" containerName="revoke-keys"
Nov 03 13:22:39.309260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.309206    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.323598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.322329    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"]
Nov 03 13:22:39.383944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.383904    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.384263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.384237    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115dd85d-bf72-4d2b-819d-84c66d23403b-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.384468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.384449    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.384606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.384588    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.384736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.384720    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.384938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.384919    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.385105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.385086    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.385255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.385238    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnd4z\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-kube-api-access-cnd4z\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.385394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.385377    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.487224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.487172    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.487434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.487248    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.487434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.487280    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnd4z\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-kube-api-access-cnd4z\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.487434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.487301    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.487434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.487368    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.487434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.487409    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115dd85d-bf72-4d2b-819d-84c66d23403b-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.487740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.487442    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.487740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.487470    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.487740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.487490    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.489118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.488545    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.489118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.488635    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.489118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.488830    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.489118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.488912    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.489118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.488940    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115dd85d-bf72-4d2b-819d-84c66d23403b-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.491661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.491639    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.491661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.491651    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.500476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.500449    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnd4z\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-kube-api-access-cnd4z\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.502657 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.502633    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.602340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.598879    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:39.657957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.657632    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:39.779963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.779927    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl" event={"ID":"3d9ab0f3-3e08-4d06-b4ac-ffc86671db84","Type":"ContainerDied","Data":"a4d120d2568bfe4d2f75cb03a86c647dc71057bc425168ebe4faab10ca339c3c"}
Nov 03 13:22:39.780275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.780254    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4d120d2568bfe4d2f75cb03a86c647dc71057bc425168ebe4faab10ca339c3c"
Nov 03 13:22:39.780565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:39.780545    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"
Nov 03 13:22:40.600851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:40.600823    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:40.801734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:40.801636    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:40.801870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:40.801808    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:40.801931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:40.801891    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:40.853316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:40.839100    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"]
Nov 03 13:22:40.860537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:40.859899    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" event={"ID":"115dd85d-bf72-4d2b-819d-84c66d23403b","Type":"ContainerStarted","Data":"6d34dff24de1b074ba32ac46d3e6a5a0f0f64be6a23e59c155650d997cb4abb5"}
Nov 03 13:22:41.313371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:41.313326    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:41.889491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:41.889448    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" event={"ID":"115dd85d-bf72-4d2b-819d-84c66d23403b","Type":"ContainerStarted","Data":"83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2"}
Nov 03 13:22:41.942234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:41.941950    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" podStartSLOduration=2.941926191 podStartE2EDuration="2.941926191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:41.912815749 +0000 UTC m=+802.990580775" watchObservedRunningTime="2026-11-03 13:22:41.941926191 +0000 UTC m=+803.019691221"
Nov 03 13:22:42.389895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.389860    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:42.581235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.578260    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"]
Nov 03 13:22:42.584347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.584060    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.586647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.586614    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"]
Nov 03 13:22:42.590720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.590702    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:22:42.590812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.590705    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:22:42.590812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.590760    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:22:42.590812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.590682    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:22:42.665242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.665121    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 13:22:42.665522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.665479    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 13:22:42.666034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.666015    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:42.674116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.674077    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce326e7b-64sbd\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.674379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.674358    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgp9f\" (UniqueName: \"kubernetes.io/projected/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-kube-api-access-cgp9f\") pod \"maas-api-revoke-sub-ce326e7b-64sbd\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.674450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.674389    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce326e7b-64sbd\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.780245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.779729    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgp9f\" (UniqueName: \"kubernetes.io/projected/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-kube-api-access-cgp9f\") pod \"maas-api-revoke-sub-ce326e7b-64sbd\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.780245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.779787    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce326e7b-64sbd\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.780245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.779967    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce326e7b-64sbd\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.781154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.781123    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce326e7b-64sbd\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.783926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.783901    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce326e7b-64sbd\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.801350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.801331    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgp9f\" (UniqueName: \"kubernetes.io/projected/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-kube-api-access-cgp9f\") pod \"maas-api-revoke-sub-ce326e7b-64sbd\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.908801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.908768    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:42.968857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:42.968723    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:43.319563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:43.319536    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:43.335241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:43.321073    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"]
Nov 03 13:22:43.550568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:43.550534    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:43.661316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:43.660798    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 13:22:43.661316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:43.660860    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 13:22:43.981379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:43.981343    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd" event={"ID":"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6","Type":"ContainerStarted","Data":"21b9f534af4e5a10e3ba238231118c53e9eb8aa148f823dd99ea3f9721c31987"}
Nov 03 13:22:43.981784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:43.981405    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd" event={"ID":"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6","Type":"ContainerStarted","Data":"e8ef1e886ea5aaa4642c832aafd49b20e116813de194b634e21cad2a821c2f51"}
Nov 03 13:22:44.000768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:44.000721    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd" podStartSLOduration=2.000706663 podStartE2EDuration="2.000706663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:43.998575195 +0000 UTC m=+805.076340218" watchObservedRunningTime="2026-11-03 13:22:44.000706663 +0000 UTC m=+805.078471698"
Nov 03 13:22:44.038238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:44.033750    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:44.670251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:44.669701    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 13:22:44.670251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:44.669778    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 13:22:44.741241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:44.739775    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:44.991146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:44.991110    5420 generic.go:356] "Generic (PLEG): container finished" podID="f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6" containerID="21b9f534af4e5a10e3ba238231118c53e9eb8aa148f823dd99ea3f9721c31987" exitCode=0
Nov 03 13:22:44.991662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:44.991151    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd" event={"ID":"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6","Type":"ContainerDied","Data":"21b9f534af4e5a10e3ba238231118c53e9eb8aa148f823dd99ea3f9721c31987"}
Nov 03 13:22:45.603656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:45.603491    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:45.664787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:45.664751    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 13:22:45.664973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:45.664828    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 13:22:45.846265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:45.844082    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:46.247383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.247352    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:46.296822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.296737    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-xks-ca\") pod \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") "
Nov 03 13:22:46.296976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.296895    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-kube-api-access-cgp9f\" (UniqueName: \"kubernetes.io/projected/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-kube-api-access-cgp9f\") pod \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") "
Nov 03 13:22:46.297050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.297024    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-openshift-ca\") pod \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\" (UID: \"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6\") "
Nov 03 13:22:46.297536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.297508    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-openshift-ca" pod "f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6" (UID: "f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:46.297730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.297714    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:46.299169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.299129    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-xks-ca" pod "f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6" (UID: "f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:46.299637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.299613    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-kube-api-access-cgp9f" pod "f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6" (UID: "f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6"). InnerVolumeSpecName "kube-api-access-cgp9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:46.399646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.399614    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:46.399646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.399647    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgp9f\" (UniqueName: \"kubernetes.io/projected/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6-kube-api-access-cgp9f\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:46.501941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.501908    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:46.658993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.658897    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 13:22:46.658993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:46.658967    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 13:22:47.008618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.008588    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"
Nov 03 13:22:47.008618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.008597    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd" event={"ID":"f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6","Type":"ContainerDied","Data":"e8ef1e886ea5aaa4642c832aafd49b20e116813de194b634e21cad2a821c2f51"}
Nov 03 13:22:47.008815 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.008657    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8ef1e886ea5aaa4642c832aafd49b20e116813de194b634e21cad2a821c2f51"
Nov 03 13:22:47.235236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.229273    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:47.465928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.465894    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"]
Nov 03 13:22:47.466852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.466832    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6" containerName="revoke-keys"
Nov 03 13:22:47.466920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.466856    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6" containerName="revoke-keys"
Nov 03 13:22:47.467030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.467020    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6" containerName="revoke-keys"
Nov 03 13:22:47.474172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.474141    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.477482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.477457    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"]
Nov 03 13:22:47.480089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.479929    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:22:47.480089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.480048    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:22:47.482027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.481374    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:22:47.516855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.516818    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:22:47.518660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.518581    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-387e50b1-t497j\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.518831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.518813    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-387e50b1-t497j\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.518901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.518861    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4q79\" (UniqueName: \"kubernetes.io/projected/4c06aeab-f308-4003-adbb-3df04b94c66b-kube-api-access-q4q79\") pod \"maas-api-revoke-sub-387e50b1-t497j\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.620021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.619975    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-387e50b1-t497j\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.620185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.620151    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-387e50b1-t497j\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.620185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.620179    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4q79\" (UniqueName: \"kubernetes.io/projected/4c06aeab-f308-4003-adbb-3df04b94c66b-kube-api-access-q4q79\") pod \"maas-api-revoke-sub-387e50b1-t497j\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.620887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.620858    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-387e50b1-t497j\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.622628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.622599    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-387e50b1-t497j\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.629678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.629655    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4q79\" (UniqueName: \"kubernetes.io/projected/4c06aeab-f308-4003-adbb-3df04b94c66b-kube-api-access-q4q79\") pod \"maas-api-revoke-sub-387e50b1-t497j\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.697859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.697825    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:47.699414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.699390    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:47.700436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.700412    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:22:47.819251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.819155    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:47.972928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:47.972892    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:48.123056 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:48.123024    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c06aeab_f308_4003_adbb_3df04b94c66b.slice/crio-939e0c020f09d30bfc1820feff453bde975f71729ac619c00b0e2a3417efbad2 WatchSource:0}: Error finding container 939e0c020f09d30bfc1820feff453bde975f71729ac619c00b0e2a3417efbad2: Status 404 returned error can't find the container with id 939e0c020f09d30bfc1820feff453bde975f71729ac619c00b0e2a3417efbad2
Nov 03 13:22:48.124270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:48.124244    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"]
Nov 03 13:22:48.590455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:48.590420    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:48.715721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:48.715695    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:22:48.768280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:48.768249    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"]
Nov 03 13:22:49.033439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:49.033360    5420 generic.go:356] "Generic (PLEG): container finished" podID="4c06aeab-f308-4003-adbb-3df04b94c66b" containerID="e020f41ae2d2cdf24c15700124366fc2ba5076529bea43b2ada5f6990648cbad" exitCode=0
Nov 03 13:22:49.033600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:49.033443    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j" event={"ID":"4c06aeab-f308-4003-adbb-3df04b94c66b","Type":"ContainerDied","Data":"e020f41ae2d2cdf24c15700124366fc2ba5076529bea43b2ada5f6990648cbad"}
Nov 03 13:22:49.033600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:49.033494    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j" event={"ID":"4c06aeab-f308-4003-adbb-3df04b94c66b","Type":"ContainerStarted","Data":"939e0c020f09d30bfc1820feff453bde975f71729ac619c00b0e2a3417efbad2"}
Nov 03 13:22:49.312872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:49.312783    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:49.604917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:49.604838    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"]
Nov 03 13:22:49.729207 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:49.729171    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:49.872618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:49.872542    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:50.048296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.048260    5420 generic.go:356] "Generic (PLEG): container finished" podID="75bdcc44-6914-4774-9b98-2f77caa51434" containerID="7d27bffe5874fb86ddbf72c3926b9f3dec917c210d09394ecc88511098123ebb" exitCode=0
Nov 03 13:22:50.048489 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.048353    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" event={"ID":"75bdcc44-6914-4774-9b98-2f77caa51434","Type":"ContainerDied","Data":"7d27bffe5874fb86ddbf72c3926b9f3dec917c210d09394ecc88511098123ebb"}
Nov 03 13:22:50.049355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.048966    5420 scope.go:122] "RemoveContainer" containerID="7d27bffe5874fb86ddbf72c3926b9f3dec917c210d09394ecc88511098123ebb"
Nov 03 13:22:50.050720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.050690    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:50.050816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.050759    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:50.050816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.050792    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:50.116831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.114356    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:50.268164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.268131    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:50.282547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.282513    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:50.311319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.311278    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:50.365879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.365831    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c06aeab-f308-4003-adbb-3df04b94c66b-kube-api-access-q4q79\" (UniqueName: \"kubernetes.io/projected/4c06aeab-f308-4003-adbb-3df04b94c66b-kube-api-access-q4q79\") pod \"4c06aeab-f308-4003-adbb-3df04b94c66b\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") "
Nov 03 13:22:50.366085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.365911    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-xks-ca\") pod \"4c06aeab-f308-4003-adbb-3df04b94c66b\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") "
Nov 03 13:22:50.366158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.366137    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-openshift-ca\") pod \"4c06aeab-f308-4003-adbb-3df04b94c66b\" (UID: \"4c06aeab-f308-4003-adbb-3df04b94c66b\") "
Nov 03 13:22:50.378329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.372727    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-openshift-ca" pod "4c06aeab-f308-4003-adbb-3df04b94c66b" (UID: "4c06aeab-f308-4003-adbb-3df04b94c66b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:50.383177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.383146    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-xks-ca" pod "4c06aeab-f308-4003-adbb-3df04b94c66b" (UID: "4c06aeab-f308-4003-adbb-3df04b94c66b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:50.383685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.383667    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c06aeab-f308-4003-adbb-3df04b94c66b-kube-api-access-q4q79" pod "4c06aeab-f308-4003-adbb-3df04b94c66b" (UID: "4c06aeab-f308-4003-adbb-3df04b94c66b"). InnerVolumeSpecName "kube-api-access-q4q79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:50.468703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.468668    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4q79\" (UniqueName: \"kubernetes.io/projected/4c06aeab-f308-4003-adbb-3df04b94c66b-kube-api-access-q4q79\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:50.468703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.468696    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:50.468703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.468707    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c06aeab-f308-4003-adbb-3df04b94c66b-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:50.993053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:50.993021    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:51.065371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:51.065342    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"
Nov 03 13:22:51.065371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:51.065350    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j" event={"ID":"4c06aeab-f308-4003-adbb-3df04b94c66b","Type":"ContainerDied","Data":"939e0c020f09d30bfc1820feff453bde975f71729ac619c00b0e2a3417efbad2"}
Nov 03 13:22:51.065589 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:51.065396    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="939e0c020f09d30bfc1820feff453bde975f71729ac619c00b0e2a3417efbad2"
Nov 03 13:22:51.067845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:51.067822    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" event={"ID":"75bdcc44-6914-4774-9b98-2f77caa51434","Type":"ContainerStarted","Data":"20030bf113a317db59c01543ccfd9f43fc8c75fd57845f66ef3a6b110d11c764"}
Nov 03 13:22:51.592678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:51.592644    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:22:51.607698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:51.607655    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:51.801727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:51.801694    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:51.802661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:51.802635    5420 patch_prober.go:29] interesting pod/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 13:22:51.802778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:51.802688    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 13:22:52.197836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:52.197808    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:52.825235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:52.824852    5420 patch_prober.go:29] interesting pod/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 13:22:52.825235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:52.824929    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 13:22:53.059251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.057711    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"]
Nov 03 13:22:53.059251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.058762    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4c06aeab-f308-4003-adbb-3df04b94c66b" containerName="revoke-keys"
Nov 03 13:22:53.059251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.058785    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c06aeab-f308-4003-adbb-3df04b94c66b" containerName="revoke-keys"
Nov 03 13:22:53.059251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.058981    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c06aeab-f308-4003-adbb-3df04b94c66b" containerName="revoke-keys"
Nov 03 13:22:53.303226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.302444    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"]
Nov 03 13:22:53.303226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.303060    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.311338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.311318    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:22:53.312400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.312376    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:22:53.409404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.407481    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6p4\" (UniqueName: \"kubernetes.io/projected/6c6e71b5-64a3-4276-8380-328c42b81016-kube-api-access-tz6p4\") pod \"maas-api-revoke-sub-494fd49f-dvwf5\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.409404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.407765    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-494fd49f-dvwf5\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.409404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.407939    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-494fd49f-dvwf5\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.410822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.410802    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:22:53.412139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.412120    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:22:53.413809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.413785    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:53.509720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.509675    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-494fd49f-dvwf5\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.509915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.509823    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-494fd49f-dvwf5\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.509915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.509883    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6p4\" (UniqueName: \"kubernetes.io/projected/6c6e71b5-64a3-4276-8380-328c42b81016-kube-api-access-tz6p4\") pod \"maas-api-revoke-sub-494fd49f-dvwf5\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.510681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.510650    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-494fd49f-dvwf5\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.513326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.513296    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-494fd49f-dvwf5\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.534055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.534025    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6p4\" (UniqueName: \"kubernetes.io/projected/6c6e71b5-64a3-4276-8380-328c42b81016-kube-api-access-tz6p4\") pod \"maas-api-revoke-sub-494fd49f-dvwf5\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.612013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.611881    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:53.713596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.713565    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:53.771535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.771300    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd" podUID="884e4c72-728e-41be-88eb-5bfc9952a3b0" containerName="payload-processing" containerID="cri-o://9a6d29bf000c5bea2f1514f9933634b7a0d4261f68649272f167185b8524c23b" gracePeriod=25
Nov 03 13:22:53.802759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.802273    5420 patch_prober.go:29] interesting pod/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 13:22:53.802759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:53.802346    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 13:22:54.570326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:54.484448    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:54.618467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:54.613299    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg" podUID="57ef622b-2fc2-4167-a663-d61a96eddca9" containerName="payload-processing" containerID="cri-o://63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771" gracePeriod=25
Nov 03 13:22:54.705752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:54.700671    5420 generic.go:356] "Generic (PLEG): container finished" podID="884e4c72-728e-41be-88eb-5bfc9952a3b0" containerID="9a6d29bf000c5bea2f1514f9933634b7a0d4261f68649272f167185b8524c23b" exitCode=0
Nov 03 13:22:54.705752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:54.700891    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd" event={"ID":"884e4c72-728e-41be-88eb-5bfc9952a3b0","Type":"ContainerDied","Data":"9a6d29bf000c5bea2f1514f9933634b7a0d4261f68649272f167185b8524c23b"}
Nov 03 13:22:54.763243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:54.760977    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:54.795089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:54.795047    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"]
Nov 03 13:22:54.805006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:54.804328    5420 patch_prober.go:29] interesting pod/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 13:22:54.805006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:54.804408    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 13:22:54.878460 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:54.855078    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6e71b5_64a3_4276_8380_328c42b81016.slice/crio-30c159e28c2fce838a679928b3a52be6d3f9ccf5e1fb1cf06da0a86bd50ff789 WatchSource:0}: Error finding container 30c159e28c2fce838a679928b3a52be6d3f9ccf5e1fb1cf06da0a86bd50ff789: Status 404 returned error can't find the container with id 30c159e28c2fce838a679928b3a52be6d3f9ccf5e1fb1cf06da0a86bd50ff789
Nov 03 13:22:54.910788 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:22:54.910734    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:22:55.028044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.027632    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:22:55.065591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.065557    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-plugins-config-volume\") pod \"884e4c72-728e-41be-88eb-5bfc9952a3b0\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") "
Nov 03 13:22:55.065774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.065618    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-routing-overlay-volume\") pod \"884e4c72-728e-41be-88eb-5bfc9952a3b0\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") "
Nov 03 13:22:55.065840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.065786    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-provider-credentials\") pod \"884e4c72-728e-41be-88eb-5bfc9952a3b0\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") "
Nov 03 13:22:55.065934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.065916    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-kube-api-access-r9szf\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-kube-api-access-r9szf\") pod \"884e4c72-728e-41be-88eb-5bfc9952a3b0\" (UID: \"884e4c72-728e-41be-88eb-5bfc9952a3b0\") "
Nov 03 13:22:55.067162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.067133    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-routing-overlay-volume" pod "884e4c72-728e-41be-88eb-5bfc9952a3b0" (UID: "884e4c72-728e-41be-88eb-5bfc9952a3b0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:55.067347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.067165    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-plugins-config-volume" pod "884e4c72-728e-41be-88eb-5bfc9952a3b0" (UID: "884e4c72-728e-41be-88eb-5bfc9952a3b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:55.141280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.120497    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:55.141280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.130821    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-provider-credentials" pod "884e4c72-728e-41be-88eb-5bfc9952a3b0" (UID: "884e4c72-728e-41be-88eb-5bfc9952a3b0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:55.146167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.146126    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-kube-api-access-r9szf" pod "884e4c72-728e-41be-88eb-5bfc9952a3b0" (UID: "884e4c72-728e-41be-88eb-5bfc9952a3b0"). InnerVolumeSpecName "kube-api-access-r9szf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:55.167738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.167710    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:55.167841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.167742    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/884e4c72-728e-41be-88eb-5bfc9952a3b0-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:55.167841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.167757    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:55.167841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.167772    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9szf\" (UniqueName: \"kubernetes.io/projected/884e4c72-728e-41be-88eb-5bfc9952a3b0-kube-api-access-r9szf\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:55.694283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.694241    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:55.710369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.710345    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"
Nov 03 13:22:55.710502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.710376    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd" event={"ID":"884e4c72-728e-41be-88eb-5bfc9952a3b0","Type":"ContainerDied","Data":"3f960f8f41b6285ef69c7afc8e3a68f808126c872931130e6b9ea2ee532a4a52"}
Nov 03 13:22:55.710502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.710440    5420 scope.go:122] "RemoveContainer" containerID="9a6d29bf000c5bea2f1514f9933634b7a0d4261f68649272f167185b8524c23b"
Nov 03 13:22:55.717004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.712851    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5" event={"ID":"6c6e71b5-64a3-4276-8380-328c42b81016","Type":"ContainerStarted","Data":"792873828a008109d9ad5699b193abb2daac9c917c23dce403062d3edae72787"}
Nov 03 13:22:55.717004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.712902    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5" event={"ID":"6c6e71b5-64a3-4276-8380-328c42b81016","Type":"ContainerStarted","Data":"30c159e28c2fce838a679928b3a52be6d3f9ccf5e1fb1cf06da0a86bd50ff789"}
Nov 03 13:22:55.740552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.740497    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5" podStartSLOduration=3.740481613 podStartE2EDuration="3.740481613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:55.737151108 +0000 UTC m=+816.814916148" watchObservedRunningTime="2026-11-03 13:22:55.740481613 +0000 UTC m=+816.818246668"
Nov 03 13:22:55.770225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.770186    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"]
Nov 03 13:22:55.797694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.797369    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866xhpvd"]
Nov 03 13:22:55.803477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.802293    5420 patch_prober.go:29] interesting pod/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 13:22:55.803477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:55.802361    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 13:22:56.185550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.185460    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:56.312736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.312664    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:56.653260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.653135    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:56.751400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.743259    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"]
Nov 03 13:22:56.751400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.744127    5420 cpu_manager.go:414] "containerMap: removing container" podUID="884e4c72-728e-41be-88eb-5bfc9952a3b0" containerName="payload-processing"
Nov 03 13:22:56.751400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.744145    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884e4c72-728e-41be-88eb-5bfc9952a3b0" containerName="payload-processing"
Nov 03 13:22:56.751400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.744358    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="884e4c72-728e-41be-88eb-5bfc9952a3b0" containerName="payload-processing"
Nov 03 13:22:56.753465 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.753446    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"]
Nov 03 13:22:56.754171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.753699    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.760654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.760636    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-384e44d183e3\""
Nov 03 13:22:56.770987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.769852    5420 generic.go:356] "Generic (PLEG): container finished" podID="6c6e71b5-64a3-4276-8380-328c42b81016" containerID="792873828a008109d9ad5699b193abb2daac9c917c23dce403062d3edae72787" exitCode=0
Nov 03 13:22:56.770987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.769885    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5" event={"ID":"6c6e71b5-64a3-4276-8380-328c42b81016","Type":"ContainerDied","Data":"792873828a008109d9ad5699b193abb2daac9c917c23dce403062d3edae72787"}
Nov 03 13:22:56.801665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.801617    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.801843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.801677    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.801987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.801968    5420 patch_prober.go:29] interesting pod/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 13:22:56.802029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.801984    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.802029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.802009    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 13:22:56.802116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.802081    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx6w7\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-kube-api-access-rx6w7\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.829327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.829293    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:56.903871 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.903774    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx6w7\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-kube-api-access-rx6w7\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.904042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.903917    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.904042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.903939    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.904175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.904124    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.904649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.904624    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.905247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.905198    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.906672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.906649    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:56.912323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:56.912294    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx6w7\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-kube-api-access-rx6w7\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:57.079677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.079645    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:57.086512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.086483    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884e4c72-728e-41be-88eb-5bfc9952a3b0" path="/var/lib/kubelet/pods/884e4c72-728e-41be-88eb-5bfc9952a3b0/volumes"
Nov 03 13:22:57.323522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.323481    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:57.365274 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:22:57.365205    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8c0281_3962_4727_866f_3990ac17c9a8.slice/crio-7080f57d99d1f2f0796a9a390537e0b7b71fb79493ba66ecb311985598913ca3 WatchSource:0}: Error finding container 7080f57d99d1f2f0796a9a390537e0b7b71fb79493ba66ecb311985598913ca3: Status 404 returned error can't find the container with id 7080f57d99d1f2f0796a9a390537e0b7b71fb79493ba66ecb311985598913ca3
Nov 03 13:22:57.366482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.366461    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"]
Nov 03 13:22:57.440903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.440873    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg" podUID="57ef622b-2fc2-4167-a663-d61a96eddca9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:22:57.632338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.631135    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:57.782578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.782520    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb" event={"ID":"be8c0281-3962-4727-866f-3990ac17c9a8","Type":"ContainerStarted","Data":"d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142"}
Nov 03 13:22:57.783161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.782593    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb" event={"ID":"be8c0281-3962-4727-866f-3990ac17c9a8","Type":"ContainerStarted","Data":"7080f57d99d1f2f0796a9a390537e0b7b71fb79493ba66ecb311985598913ca3"}
Nov 03 13:22:57.783161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.782903    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:22:57.808406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.808374    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:57.809221 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.809156    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb" podStartSLOduration=1.809140861 podStartE2EDuration="1.809140861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:57.803120617 +0000 UTC m=+818.880885645" watchObservedRunningTime="2026-11-03 13:22:57.809140861 +0000 UTC m=+818.886905888"
Nov 03 13:22:57.809324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.809191    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:57.809954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.809927    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:22:57.962053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:57.962029    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:58.018543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.018512    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6e71b5-64a3-4276-8380-328c42b81016-kube-api-access-tz6p4\" (UniqueName: \"kubernetes.io/projected/6c6e71b5-64a3-4276-8380-328c42b81016-kube-api-access-tz6p4\") pod \"6c6e71b5-64a3-4276-8380-328c42b81016\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") "
Nov 03 13:22:58.018746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.018555    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-openshift-ca\") pod \"6c6e71b5-64a3-4276-8380-328c42b81016\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") "
Nov 03 13:22:58.018746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.018701    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-xks-ca\") pod \"6c6e71b5-64a3-4276-8380-328c42b81016\" (UID: \"6c6e71b5-64a3-4276-8380-328c42b81016\") "
Nov 03 13:22:58.019612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.019574    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-openshift-ca" pod "6c6e71b5-64a3-4276-8380-328c42b81016" (UID: "6c6e71b5-64a3-4276-8380-328c42b81016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:22:58.020360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.020337    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:58.034299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.034262    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6e71b5-64a3-4276-8380-328c42b81016-kube-api-access-tz6p4" pod "6c6e71b5-64a3-4276-8380-328c42b81016" (UID: "6c6e71b5-64a3-4276-8380-328c42b81016"). InnerVolumeSpecName "kube-api-access-tz6p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:22:58.034406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.034296    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-xks-ca" pod "6c6e71b5-64a3-4276-8380-328c42b81016" (UID: "6c6e71b5-64a3-4276-8380-328c42b81016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:22:58.121375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.121343    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6e71b5-64a3-4276-8380-328c42b81016-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:58.121375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.121370    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz6p4\" (UniqueName: \"kubernetes.io/projected/6c6e71b5-64a3-4276-8380-328c42b81016-kube-api-access-tz6p4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:22:58.297696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.297667    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:58.796837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.795880    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"
Nov 03 13:22:58.796837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.795925    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5" event={"ID":"6c6e71b5-64a3-4276-8380-328c42b81016","Type":"ContainerDied","Data":"30c159e28c2fce838a679928b3a52be6d3f9ccf5e1fb1cf06da0a86bd50ff789"}
Nov 03 13:22:58.796837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.795973    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30c159e28c2fce838a679928b3a52be6d3f9ccf5e1fb1cf06da0a86bd50ff789"
Nov 03 13:22:58.831037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:58.831006    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:59.070912 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:59.070814    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:59.781747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:22:59.781716    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:00.430046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:00.430013    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:01.105344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:01.105308    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:01.731274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:01.730615    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:02.403743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:02.403707    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:02.923956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:02.923920    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:03.069192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:03.069148    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:03.479882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:03.479843    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:03.811889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:03.811790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:04.017562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:04.017518    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:04.634267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:04.634226    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:04.982743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:04.982711    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:05.716778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.712269    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"]
Nov 03 13:23:05.716778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.713284    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6c6e71b5-64a3-4276-8380-328c42b81016" containerName="revoke-keys"
Nov 03 13:23:05.716778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.713303    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6e71b5-64a3-4276-8380-328c42b81016" containerName="revoke-keys"
Nov 03 13:23:05.716778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.713509    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6e71b5-64a3-4276-8380-328c42b81016" containerName="revoke-keys"
Nov 03 13:23:05.718778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.718754    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:05.725069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.725045    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:23:05.725457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.725410    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:23:05.735309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.735288    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:23:05.735558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.735542    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:23:05.812563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.812530    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"]
Nov 03 13:23:05.859634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.859602    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef380069-zzqvf\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:05.859841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.859766    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef380069-zzqvf\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:05.859913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.859898    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4tl\" (UniqueName: \"kubernetes.io/projected/0508373e-4df4-462d-815a-e2dccd53388e-kube-api-access-nf4tl\") pod \"maas-api-revoke-sub-ef380069-zzqvf\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:05.932101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.932070    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:05.961456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.961416    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef380069-zzqvf\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:05.961629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.961573    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4tl\" (UniqueName: \"kubernetes.io/projected/0508373e-4df4-462d-815a-e2dccd53388e-kube-api-access-nf4tl\") pod \"maas-api-revoke-sub-ef380069-zzqvf\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:05.961711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.961689    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef380069-zzqvf\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:05.962418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.962397    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef380069-zzqvf\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:05.964224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.964190    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef380069-zzqvf\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:05.971162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:05.971108    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4tl\" (UniqueName: \"kubernetes.io/projected/0508373e-4df4-462d-815a-e2dccd53388e-kube-api-access-nf4tl\") pod \"maas-api-revoke-sub-ef380069-zzqvf\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:06.113418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:06.113375    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:06.627369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:06.627139    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:06.683653 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:06.675950    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"]
Nov 03 13:23:06.703137 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:06.700633    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0508373e_4df4_462d_815a_e2dccd53388e.slice/crio-dd77a65059710bdd67d348a505f3fd4eae08cebe0e100d48fe500898707221ab WatchSource:0}: Error finding container dd77a65059710bdd67d348a505f3fd4eae08cebe0e100d48fe500898707221ab: Status 404 returned error can't find the container with id dd77a65059710bdd67d348a505f3fd4eae08cebe0e100d48fe500898707221ab
Nov 03 13:23:06.899930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:06.899662    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf" event={"ID":"0508373e-4df4-462d-815a-e2dccd53388e","Type":"ContainerStarted","Data":"dd77a65059710bdd67d348a505f3fd4eae08cebe0e100d48fe500898707221ab"}
Nov 03 13:23:07.456235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:07.455294    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg" podUID="57ef622b-2fc2-4167-a663-d61a96eddca9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:23:07.542288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:07.526715    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:07.909340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:07.909255    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf" event={"ID":"0508373e-4df4-462d-815a-e2dccd53388e","Type":"ContainerStarted","Data":"6f5d3594c18cb5ca419a59b9de4adb1c1466768df3a19db1553f06facfa464d5"}
Nov 03 13:23:07.928012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:07.927954    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf" podStartSLOduration=2.927941777 podStartE2EDuration="2.927941777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:07.925319809 +0000 UTC m=+829.003084836" watchObservedRunningTime="2026-11-03 13:23:07.927941777 +0000 UTC m=+829.005706803"
Nov 03 13:23:08.191623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:08.191551    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:08.794636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:08.794600    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:08.800916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:08.800894    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:23:08.871061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:08.870984    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"]
Nov 03 13:23:08.922433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:08.922401    5420 generic.go:356] "Generic (PLEG): container finished" podID="0508373e-4df4-462d-815a-e2dccd53388e" containerID="6f5d3594c18cb5ca419a59b9de4adb1c1466768df3a19db1553f06facfa464d5" exitCode=0
Nov 03 13:23:08.922921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:08.922478    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf" event={"ID":"0508373e-4df4-462d-815a-e2dccd53388e","Type":"ContainerDied","Data":"6f5d3594c18cb5ca419a59b9de4adb1c1466768df3a19db1553f06facfa464d5"}
Nov 03 13:23:09.409188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.409158    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:09.738933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.738895    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"]
Nov 03 13:23:09.745495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.745468    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:09.750721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.750690    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"]
Nov 03 13:23:09.868825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.868794    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzfk4\" (UniqueName: \"kubernetes.io/projected/195f6c59-4d56-4060-b25e-9c6993cf8977-kube-api-access-jzfk4\") pod \"maas-api-revoke-sub-092ca36a-f999l\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:09.869021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.868978    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-092ca36a-f999l\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:09.869096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.869020    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-092ca36a-f999l\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:09.970664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.970625    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-092ca36a-f999l\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:09.971040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.970693    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-092ca36a-f999l\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:09.971040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.970925    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzfk4\" (UniqueName: \"kubernetes.io/projected/195f6c59-4d56-4060-b25e-9c6993cf8977-kube-api-access-jzfk4\") pod \"maas-api-revoke-sub-092ca36a-f999l\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:09.971602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.971575    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-092ca36a-f999l\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:09.973307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.973283    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-092ca36a-f999l\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:09.979466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:09.979432    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzfk4\" (UniqueName: \"kubernetes.io/projected/195f6c59-4d56-4060-b25e-9c6993cf8977-kube-api-access-jzfk4\") pod \"maas-api-revoke-sub-092ca36a-f999l\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:10.063931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.063895    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:10.096168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.096148    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:10.130059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.130033    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:10.174007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.173972    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0508373e-4df4-462d-815a-e2dccd53388e-kube-api-access-nf4tl\" (UniqueName: \"kubernetes.io/projected/0508373e-4df4-462d-815a-e2dccd53388e-kube-api-access-nf4tl\") pod \"0508373e-4df4-462d-815a-e2dccd53388e\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") "
Nov 03 13:23:10.174197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.174047    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-openshift-ca\") pod \"0508373e-4df4-462d-815a-e2dccd53388e\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") "
Nov 03 13:23:10.174197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.174140    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-xks-ca\") pod \"0508373e-4df4-462d-815a-e2dccd53388e\" (UID: \"0508373e-4df4-462d-815a-e2dccd53388e\") "
Nov 03 13:23:10.175194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.174940    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-openshift-ca" pod "0508373e-4df4-462d-815a-e2dccd53388e" (UID: "0508373e-4df4-462d-815a-e2dccd53388e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:10.178961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.178928    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:10.185739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.185702    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-xks-ca" pod "0508373e-4df4-462d-815a-e2dccd53388e" (UID: "0508373e-4df4-462d-815a-e2dccd53388e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:10.185869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.185826    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0508373e-4df4-462d-815a-e2dccd53388e-kube-api-access-nf4tl" pod "0508373e-4df4-462d-815a-e2dccd53388e" (UID: "0508373e-4df4-462d-815a-e2dccd53388e"). InnerVolumeSpecName "kube-api-access-nf4tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:10.279875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.279844    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf4tl\" (UniqueName: \"kubernetes.io/projected/0508373e-4df4-462d-815a-e2dccd53388e-kube-api-access-nf4tl\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:10.279875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.279871    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0508373e-4df4-462d-815a-e2dccd53388e-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:10.347512 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:10.347486    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195f6c59_4d56_4060_b25e_9c6993cf8977.slice/crio-65b30f9d41a1d9a85c41a4c14f43e7c31507ab2a525a52ba58a12ab55c2ae077 WatchSource:0}: Error finding container 65b30f9d41a1d9a85c41a4c14f43e7c31507ab2a525a52ba58a12ab55c2ae077: Status 404 returned error can't find the container with id 65b30f9d41a1d9a85c41a4c14f43e7c31507ab2a525a52ba58a12ab55c2ae077
Nov 03 13:23:10.349677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.349651    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"]
Nov 03 13:23:10.780855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.780817    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"]
Nov 03 13:23:10.807288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.807251    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:10.941002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.940926    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"
Nov 03 13:23:10.941002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.940967    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf" event={"ID":"0508373e-4df4-462d-815a-e2dccd53388e","Type":"ContainerDied","Data":"dd77a65059710bdd67d348a505f3fd4eae08cebe0e100d48fe500898707221ab"}
Nov 03 13:23:10.941254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.941015    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd77a65059710bdd67d348a505f3fd4eae08cebe0e100d48fe500898707221ab"
Nov 03 13:23:10.941254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.941110    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:10.942719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.942680    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l" event={"ID":"195f6c59-4d56-4060-b25e-9c6993cf8977","Type":"ContainerStarted","Data":"267104f3ddc52fb4ad230a73f37cd322cf867149a4abdb9797ed5a3782ff4618"}
Nov 03 13:23:10.942855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.942724    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l" event={"ID":"195f6c59-4d56-4060-b25e-9c6993cf8977","Type":"ContainerStarted","Data":"65b30f9d41a1d9a85c41a4c14f43e7c31507ab2a525a52ba58a12ab55c2ae077"}
Nov 03 13:23:10.964652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:10.964606    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l" podStartSLOduration=1.9645941329999999 podStartE2EDuration="1.964594133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:10.959023829 +0000 UTC m=+832.036788856" watchObservedRunningTime="2026-11-03 13:23:10.964594133 +0000 UTC m=+832.042359158"
Nov 03 13:23:11.083690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:11.083658    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:11.324354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:11.324328    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:11.533202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:11.533106    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:11.953001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:11.952970    5420 generic.go:356] "Generic (PLEG): container finished" podID="195f6c59-4d56-4060-b25e-9c6993cf8977" containerID="267104f3ddc52fb4ad230a73f37cd322cf867149a4abdb9797ed5a3782ff4618" exitCode=0
Nov 03 13:23:11.953165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:11.953051    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l" event={"ID":"195f6c59-4d56-4060-b25e-9c6993cf8977","Type":"ContainerDied","Data":"267104f3ddc52fb4ad230a73f37cd322cf867149a4abdb9797ed5a3782ff4618"}
Nov 03 13:23:12.184105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:12.184072    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:12.803934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:12.803900    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:13.119461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.119434    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:13.220999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.220965    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-openshift-ca\") pod \"195f6c59-4d56-4060-b25e-9c6993cf8977\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") "
Nov 03 13:23:13.221473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.221013    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-xks-ca\") pod \"195f6c59-4d56-4060-b25e-9c6993cf8977\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") "
Nov 03 13:23:13.221473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.221241    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195f6c59-4d56-4060-b25e-9c6993cf8977-kube-api-access-jzfk4\" (UniqueName: \"kubernetes.io/projected/195f6c59-4d56-4060-b25e-9c6993cf8977-kube-api-access-jzfk4\") pod \"195f6c59-4d56-4060-b25e-9c6993cf8977\" (UID: \"195f6c59-4d56-4060-b25e-9c6993cf8977\") "
Nov 03 13:23:13.221473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.221311    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-openshift-ca" pod "195f6c59-4d56-4060-b25e-9c6993cf8977" (UID: "195f6c59-4d56-4060-b25e-9c6993cf8977"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:13.221805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.221781    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:13.223134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.223111    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-xks-ca" pod "195f6c59-4d56-4060-b25e-9c6993cf8977" (UID: "195f6c59-4d56-4060-b25e-9c6993cf8977"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:13.223314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.223261    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195f6c59-4d56-4060-b25e-9c6993cf8977-kube-api-access-jzfk4" pod "195f6c59-4d56-4060-b25e-9c6993cf8977" (UID: "195f6c59-4d56-4060-b25e-9c6993cf8977"). InnerVolumeSpecName "kube-api-access-jzfk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:13.323042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.323012    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195f6c59-4d56-4060-b25e-9c6993cf8977-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:13.323042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.323042    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzfk4\" (UniqueName: \"kubernetes.io/projected/195f6c59-4d56-4060-b25e-9c6993cf8977-kube-api-access-jzfk4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:13.402510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.402402    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:13.864269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.864237    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:13.872064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:13.872017    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng" podUID="f3080298-1815-4006-ae0e-fcd845f65ee7" containerName="payload-processing" containerID="cri-o://946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885" gracePeriod=25
Nov 03 13:23:14.029171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.029114    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l" event={"ID":"195f6c59-4d56-4060-b25e-9c6993cf8977","Type":"ContainerDied","Data":"65b30f9d41a1d9a85c41a4c14f43e7c31507ab2a525a52ba58a12ab55c2ae077"}
Nov 03 13:23:14.029623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.029606    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b30f9d41a1d9a85c41a4c14f43e7c31507ab2a525a52ba58a12ab55c2ae077"
Nov 03 13:23:14.029738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.029558    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"
Nov 03 13:23:14.118031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.115420    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:14.262036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.262014    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:23:14.335793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.335763    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-provider-credentials\") pod \"f3080298-1815-4006-ae0e-fcd845f65ee7\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") "
Nov 03 13:23:14.336010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.335933    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-routing-overlay-volume\") pod \"f3080298-1815-4006-ae0e-fcd845f65ee7\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") "
Nov 03 13:23:14.336273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.336102    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-plugins-config-volume\") pod \"f3080298-1815-4006-ae0e-fcd845f65ee7\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") "
Nov 03 13:23:14.336273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.336195    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-kube-api-access-vpdgk\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-kube-api-access-vpdgk\") pod \"f3080298-1815-4006-ae0e-fcd845f65ee7\" (UID: \"f3080298-1815-4006-ae0e-fcd845f65ee7\") "
Nov 03 13:23:14.336501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.336478    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-routing-overlay-volume" pod "f3080298-1815-4006-ae0e-fcd845f65ee7" (UID: "f3080298-1815-4006-ae0e-fcd845f65ee7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:14.337006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.336878    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:14.337179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.337154    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-plugins-config-volume" pod "f3080298-1815-4006-ae0e-fcd845f65ee7" (UID: "f3080298-1815-4006-ae0e-fcd845f65ee7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:14.338755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.338547    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-kube-api-access-vpdgk" pod "f3080298-1815-4006-ae0e-fcd845f65ee7" (UID: "f3080298-1815-4006-ae0e-fcd845f65ee7"). InnerVolumeSpecName "kube-api-access-vpdgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:14.342249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.340383    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-provider-credentials" pod "f3080298-1815-4006-ae0e-fcd845f65ee7" (UID: "f3080298-1815-4006-ae0e-fcd845f65ee7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:14.438617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.438538    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3080298-1815-4006-ae0e-fcd845f65ee7-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:14.438617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.438566    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpdgk\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-kube-api-access-vpdgk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:14.438617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.438576    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3080298-1815-4006-ae0e-fcd845f65ee7-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:14.728181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.728146    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:14.863995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.863971    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:23:14.953033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.953006    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ef622b-2fc2-4167-a663-d61a96eddca9-kube-api-access-wnn7c\" (UniqueName: \"kubernetes.io/projected/57ef622b-2fc2-4167-a663-d61a96eddca9-kube-api-access-wnn7c\") pod \"57ef622b-2fc2-4167-a663-d61a96eddca9\" (UID: \"57ef622b-2fc2-4167-a663-d61a96eddca9\") "
Nov 03 13:23:14.953222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.953137    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ef622b-2fc2-4167-a663-d61a96eddca9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ef622b-2fc2-4167-a663-d61a96eddca9-plugins-config-volume\") pod \"57ef622b-2fc2-4167-a663-d61a96eddca9\" (UID: \"57ef622b-2fc2-4167-a663-d61a96eddca9\") "
Nov 03 13:23:14.953589 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.953558    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ef622b-2fc2-4167-a663-d61a96eddca9-plugins-config-volume" pod "57ef622b-2fc2-4167-a663-d61a96eddca9" (UID: "57ef622b-2fc2-4167-a663-d61a96eddca9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:14.953839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.953818    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ef622b-2fc2-4167-a663-d61a96eddca9-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:14.955429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:14.955402    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ef622b-2fc2-4167-a663-d61a96eddca9-kube-api-access-wnn7c" pod "57ef622b-2fc2-4167-a663-d61a96eddca9" (UID: "57ef622b-2fc2-4167-a663-d61a96eddca9"). InnerVolumeSpecName "kube-api-access-wnn7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:15.037889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.037857    5420 generic.go:356] "Generic (PLEG): container finished" podID="f3080298-1815-4006-ae0e-fcd845f65ee7" containerID="946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885" exitCode=0
Nov 03 13:23:15.038062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.037906    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"
Nov 03 13:23:15.038062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.037946    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng" event={"ID":"f3080298-1815-4006-ae0e-fcd845f65ee7","Type":"ContainerDied","Data":"946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885"}
Nov 03 13:23:15.038062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.038006    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng" event={"ID":"f3080298-1815-4006-ae0e-fcd845f65ee7","Type":"ContainerDied","Data":"b1bfbeb458c172369d0c3974d0b727360ee29bc275f7d04b6fbd862b92f6887b"}
Nov 03 13:23:15.038062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.038035    5420 scope.go:122] "RemoveContainer" containerID="946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885"
Nov 03 13:23:15.039642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.039619    5420 generic.go:356] "Generic (PLEG): container finished" podID="57ef622b-2fc2-4167-a663-d61a96eddca9" containerID="63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771" exitCode=0
Nov 03 13:23:15.039762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.039656    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"
Nov 03 13:23:15.039762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.039695    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg" event={"ID":"57ef622b-2fc2-4167-a663-d61a96eddca9","Type":"ContainerDied","Data":"63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771"}
Nov 03 13:23:15.039762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.039748    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg" event={"ID":"57ef622b-2fc2-4167-a663-d61a96eddca9","Type":"ContainerDied","Data":"9b9776cff959c7f32aa40f714d1b54f78e15f936f9e84b3414261a0198d4ab54"}
Nov 03 13:23:15.052708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.052688    5420 scope.go:122] "RemoveContainer" containerID="946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885"
Nov 03 13:23:15.053108 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:23:15.053085    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885\": container with ID starting with 946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885 not found: ID does not exist" containerID="946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885"
Nov 03 13:23:15.053198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.053119    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885"} err="failed to get container status \"946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885\": rpc error: code = NotFound desc = could not find container \"946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885\": container with ID starting with 946ea67db3d67d9732ee5744c8729bb958e25d8e1d27e664da44c2d63fb0b885 not found: ID does not exist"
Nov 03 13:23:15.053198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.053135    5420 scope.go:122] "RemoveContainer" containerID="63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771"
Nov 03 13:23:15.054666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.054645    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnn7c\" (UniqueName: \"kubernetes.io/projected/57ef622b-2fc2-4167-a663-d61a96eddca9-kube-api-access-wnn7c\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:15.063069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.062898    5420 scope.go:122] "RemoveContainer" containerID="63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771"
Nov 03 13:23:15.063173 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:23:15.063155    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771\": container with ID starting with 63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771 not found: ID does not exist" containerID="63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771"
Nov 03 13:23:15.063234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.063180    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771"} err="failed to get container status \"63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771\": rpc error: code = NotFound desc = could not find container \"63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771\": container with ID starting with 63457ef9bbf0ae618715ce489a99bb6f050c06352f2bd071b83d25d93cb17771 not found: ID does not exist"
Nov 03 13:23:15.073590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.073568    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"]
Nov 03 13:23:15.085400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.085367    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17adbf-dc946b59f-9v2wg"]
Nov 03 13:23:15.087179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.087161    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"]
Nov 03 13:23:15.091524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.091505    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6d75sg9ng"]
Nov 03 13:23:15.290270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.290236    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:15.782038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.781978    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj" podUID="41b89cf0-3d7f-486b-bd21-fe8193a2ea89" containerName="payload-processing" containerID="cri-o://e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e" gracePeriod=25
Nov 03 13:23:15.897721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:15.897686    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:16.491002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:16.490958    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:16.558570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:16.558525    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj" podUID="41b89cf0-3d7f-486b-bd21-fe8193a2ea89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:23:16.859246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:16.859142    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:23:17.085855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:17.085816    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ef622b-2fc2-4167-a663-d61a96eddca9" path="/var/lib/kubelet/pods/57ef622b-2fc2-4167-a663-d61a96eddca9/volumes"
Nov 03 13:23:17.086264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:17.086248    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3080298-1815-4006-ae0e-fcd845f65ee7" path="/var/lib/kubelet/pods/f3080298-1815-4006-ae0e-fcd845f65ee7/volumes"
Nov 03 13:23:17.180742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:17.180663    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:19.410740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.410696    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:19.736406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.736358    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:19.787268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.775747    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"]
Nov 03 13:23:19.787268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.776150    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy" containerID="cri-o://a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6" gracePeriod=30
Nov 03 13:23:19.788970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.788928    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:19.867447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.860929    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:19.867447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.861409    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 13:23:19.867447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.862130    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 03 13:23:19.867447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.862505    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Nov 03 13:23:19.869197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:19.869159    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 13:23:20.269656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.269537    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:20.310723 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.310691    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"]
Nov 03 13:23:20.312151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.312115    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0508373e-4df4-462d-815a-e2dccd53388e" containerName="revoke-keys"
Nov 03 13:23:20.312338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.312323    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0508373e-4df4-462d-815a-e2dccd53388e" containerName="revoke-keys"
Nov 03 13:23:20.313241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.313206    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f3080298-1815-4006-ae0e-fcd845f65ee7" containerName="payload-processing"
Nov 03 13:23:20.313382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.313367    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3080298-1815-4006-ae0e-fcd845f65ee7" containerName="payload-processing"
Nov 03 13:23:20.313514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.313501    5420 cpu_manager.go:414] "containerMap: removing container" podUID="57ef622b-2fc2-4167-a663-d61a96eddca9" containerName="payload-processing"
Nov 03 13:23:20.313618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.313605    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ef622b-2fc2-4167-a663-d61a96eddca9" containerName="payload-processing"
Nov 03 13:23:20.313743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.313730    5420 cpu_manager.go:414] "containerMap: removing container" podUID="195f6c59-4d56-4060-b25e-9c6993cf8977" containerName="revoke-keys"
Nov 03 13:23:20.313832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.313820    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195f6c59-4d56-4060-b25e-9c6993cf8977" containerName="revoke-keys"
Nov 03 13:23:20.314099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.314081    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ef622b-2fc2-4167-a663-d61a96eddca9" containerName="payload-processing"
Nov 03 13:23:20.314245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.314205    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="195f6c59-4d56-4060-b25e-9c6993cf8977" containerName="revoke-keys"
Nov 03 13:23:20.325748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.325712    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0508373e-4df4-462d-815a-e2dccd53388e" containerName="revoke-keys"
Nov 03 13:23:20.325966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.325950    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3080298-1815-4006-ae0e-fcd845f65ee7" containerName="payload-processing"
Nov 03 13:23:20.359641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.358383    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:20.375181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.375147    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:20.384110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.384086    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.406688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.406568    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:23:20.406688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.406642    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:23:20.407047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.407028    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:23:20.407300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.407274    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:23:20.426181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.426151    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"]
Nov 03 13:23:20.437235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.432913    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:20.515161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.515126    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227501c7-rjk9n\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.515550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.515517    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227501c7-rjk9n\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.523696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.523616    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckc76\" (UniqueName: \"kubernetes.io/projected/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-kube-api-access-ckc76\") pod \"maas-api-revoke-sub-227501c7-rjk9n\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.628331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.628290    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227501c7-rjk9n\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.628639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.628618    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227501c7-rjk9n\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.628788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.628770    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckc76\" (UniqueName: \"kubernetes.io/projected/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-kube-api-access-ckc76\") pod \"maas-api-revoke-sub-227501c7-rjk9n\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.630168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.630143    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227501c7-rjk9n\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.633350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.633306    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227501c7-rjk9n\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.644256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.643922    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckc76\" (UniqueName: \"kubernetes.io/projected/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-kube-api-access-ckc76\") pod \"maas-api-revoke-sub-227501c7-rjk9n\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.717042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.716982    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:20.826595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.826474    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:20.961361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:20.961311    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:21.040410 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:21.040378    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eff82c1_72c1_4fb8_8f53_3c1aa2dbf333.slice/crio-f878c78157f187ec51db237f087b12c23f457eae5f8ae0b78e4c6e50da5338aa WatchSource:0}: Error finding container f878c78157f187ec51db237f087b12c23f457eae5f8ae0b78e4c6e50da5338aa: Status 404 returned error can't find the container with id f878c78157f187ec51db237f087b12c23f457eae5f8ae0b78e4c6e50da5338aa
Nov 03 13:23:21.043824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:21.043794    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"]
Nov 03 13:23:21.139914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:21.139881    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:21.204054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:21.204023    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n" event={"ID":"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333","Type":"ContainerStarted","Data":"f878c78157f187ec51db237f087b12c23f457eae5f8ae0b78e4c6e50da5338aa"}
Nov 03 13:23:21.631030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:21.630996    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:22.213517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:22.213488    5420 generic.go:356] "Generic (PLEG): container finished" podID="8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333" containerID="37730fd654700ac89cb948bcd63a4fa4b1deb76f8afcbb940780d724761a8369" exitCode=0
Nov 03 13:23:22.213667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:22.213564    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n" event={"ID":"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333","Type":"ContainerDied","Data":"37730fd654700ac89cb948bcd63a4fa4b1deb76f8afcbb940780d724761a8369"}
Nov 03 13:23:22.224273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:22.224249    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:22.873689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:22.873657    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:23.381401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.381380    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:23.479611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.479539    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-xks-ca\") pod \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") "
Nov 03 13:23:23.479796 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.479774    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-openshift-ca\") pod \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") "
Nov 03 13:23:23.479901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.479847    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-kube-api-access-ckc76\" (UniqueName: \"kubernetes.io/projected/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-kube-api-access-ckc76\") pod \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\" (UID: \"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333\") "
Nov 03 13:23:23.480168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.480138    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-openshift-ca" pod "8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333" (UID: "8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:23.480324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.480307    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:23.481621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.481596    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-xks-ca" pod "8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333" (UID: "8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:23.481859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.481842    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-kube-api-access-ckc76" pod "8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333" (UID: "8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333"). InnerVolumeSpecName "kube-api-access-ckc76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:23.509946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.509919    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:23.581754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.581722    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckc76\" (UniqueName: \"kubernetes.io/projected/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-kube-api-access-ckc76\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:23.581754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:23.581752    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:24.233272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:24.233206    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"
Nov 03 13:23:24.233272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:24.233249    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n" event={"ID":"8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333","Type":"ContainerDied","Data":"f878c78157f187ec51db237f087b12c23f457eae5f8ae0b78e4c6e50da5338aa"}
Nov 03 13:23:24.233703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:24.233303    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f878c78157f187ec51db237f087b12c23f457eae5f8ae0b78e4c6e50da5338aa"
Nov 03 13:23:24.287150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:24.287121    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:24.731605 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:23:24.731568    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:23:24.924905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:24.924874    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:25.114417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.114363    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:25.229291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.229269    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:23:25.245053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.245021    5420 generic.go:356] "Generic (PLEG): container finished" podID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerID="a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6" exitCode=0
Nov 03 13:23:25.245412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.245079    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"
Nov 03 13:23:25.245412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.245107    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" event={"ID":"d2cb690a-f1c0-4871-8c1b-203584b7645e","Type":"ContainerDied","Data":"a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6"}
Nov 03 13:23:25.245412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.245173    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn" event={"ID":"d2cb690a-f1c0-4871-8c1b-203584b7645e","Type":"ContainerDied","Data":"6e039d83c9386448be1599e91e8ef8ad745a9849263e718b7136d5cb9bbedb0f"}
Nov 03 13:23:25.245412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.245205    5420 scope.go:122] "RemoveContainer" containerID="a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6"
Nov 03 13:23:25.259000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.258982    5420 scope.go:122] "RemoveContainer" containerID="711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718"
Nov 03 13:23:25.273593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.273568    5420 scope.go:122] "RemoveContainer" containerID="a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6"
Nov 03 13:23:25.274201 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:23:25.274121    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6\": container with ID starting with a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6 not found: ID does not exist" containerID="a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6"
Nov 03 13:23:25.274334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.274250    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6"} err="failed to get container status \"a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6\": rpc error: code = NotFound desc = could not find container \"a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6\": container with ID starting with a27356e53ebeba7bb98b65c63f1265e406b951f9af01c2f0d319e21b477e2fc6 not found: ID does not exist"
Nov 03 13:23:25.274334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.274312    5420 scope.go:122] "RemoveContainer" containerID="711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718"
Nov 03 13:23:25.274675 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:23:25.274649    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718\": container with ID starting with 711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718 not found: ID does not exist" containerID="711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718"
Nov 03 13:23:25.274780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.274684    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718"} err="failed to get container status \"711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718\": rpc error: code = NotFound desc = could not find container \"711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718\": container with ID starting with 711737ad4a14ca3450969b38de612b02a49fe01420a56cbb3caae7a42f0e2718 not found: ID does not exist"
Nov 03 13:23:25.316169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316096    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-podinfo\") pod \"d2cb690a-f1c0-4871-8c1b-203584b7645e\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") "
Nov 03 13:23:25.316305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316258    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-certs\") pod \"d2cb690a-f1c0-4871-8c1b-203584b7645e\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") "
Nov 03 13:23:25.316364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316312    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-token\") pod \"d2cb690a-f1c0-4871-8c1b-203584b7645e\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") "
Nov 03 13:23:25.316430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316360    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-data\") pod \"d2cb690a-f1c0-4871-8c1b-203584b7645e\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") "
Nov 03 13:23:25.316430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316421    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-credential-socket\") pod \"d2cb690a-f1c0-4871-8c1b-203584b7645e\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") "
Nov 03 13:23:25.316531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316511    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-socket\") pod \"d2cb690a-f1c0-4871-8c1b-203584b7645e\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") "
Nov 03 13:23:25.316579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316546    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-certs" pod "d2cb690a-f1c0-4871-8c1b-203584b7645e" (UID: "d2cb690a-f1c0-4871-8c1b-203584b7645e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:23:25.316627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316597    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-kube-api-access-62whp\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-kube-api-access-62whp\") pod \"d2cb690a-f1c0-4871-8c1b-203584b7645e\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") "
Nov 03 13:23:25.316682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316641    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-data" pod "d2cb690a-f1c0-4871-8c1b-203584b7645e" (UID: "d2cb690a-f1c0-4871-8c1b-203584b7645e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:23:25.316682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316677    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-envoy\") pod \"d2cb690a-f1c0-4871-8c1b-203584b7645e\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") "
Nov 03 13:23:25.316798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316713    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2cb690a-f1c0-4871-8c1b-203584b7645e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb690a-f1c0-4871-8c1b-203584b7645e-istiod-ca-cert\") pod \"d2cb690a-f1c0-4871-8c1b-203584b7645e\" (UID: \"d2cb690a-f1c0-4871-8c1b-203584b7645e\") "
Nov 03 13:23:25.316798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316744    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-credential-socket" pod "d2cb690a-f1c0-4871-8c1b-203584b7645e" (UID: "d2cb690a-f1c0-4871-8c1b-203584b7645e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:23:25.316798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.316788    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-socket" pod "d2cb690a-f1c0-4871-8c1b-203584b7645e" (UID: "d2cb690a-f1c0-4871-8c1b-203584b7645e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:23:25.317118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.317084    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cb690a-f1c0-4871-8c1b-203584b7645e-istiod-ca-cert" pod "d2cb690a-f1c0-4871-8c1b-203584b7645e" (UID: "d2cb690a-f1c0-4871-8c1b-203584b7645e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:25.317324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.317259    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:25.317324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.317276    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:25.317324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.317286    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:25.317324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.317298    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:25.317324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.317314    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cb690a-f1c0-4871-8c1b-203584b7645e-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:25.319004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.318979    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-envoy" pod "d2cb690a-f1c0-4871-8c1b-203584b7645e" (UID: "d2cb690a-f1c0-4871-8c1b-203584b7645e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:23:25.319148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.319123    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-podinfo" pod "d2cb690a-f1c0-4871-8c1b-203584b7645e" (UID: "d2cb690a-f1c0-4871-8c1b-203584b7645e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:23:25.319358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.319329    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-kube-api-access-62whp" pod "d2cb690a-f1c0-4871-8c1b-203584b7645e" (UID: "d2cb690a-f1c0-4871-8c1b-203584b7645e"). InnerVolumeSpecName "kube-api-access-62whp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:25.319443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.319368    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-token" pod "d2cb690a-f1c0-4871-8c1b-203584b7645e" (UID: "d2cb690a-f1c0-4871-8c1b-203584b7645e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:25.418494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.418462    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:25.418494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.418492    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62whp\" (UniqueName: \"kubernetes.io/projected/d2cb690a-f1c0-4871-8c1b-203584b7645e-kube-api-access-62whp\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:25.418671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.418503    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:25.418671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.418512    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cb690a-f1c0-4871-8c1b-203584b7645e-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:25.573124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.573058    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"]
Nov 03 13:23:25.578019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.577993    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-db71f4-openshift-default-5c5dc9b98f-knmrn"]
Nov 03 13:23:25.716804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:25.716773    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:26.225588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.225558    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:26.240433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.240402    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"]
Nov 03 13:23:26.240997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.240984    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy"
Nov 03 13:23:26.241048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.240999    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy"
Nov 03 13:23:26.241048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.241021    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy"
Nov 03 13:23:26.241048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.241027    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy"
Nov 03 13:23:26.241048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.241039    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333" containerName="revoke-keys"
Nov 03 13:23:26.241048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.241044    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333" containerName="revoke-keys"
Nov 03 13:23:26.241201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.241125    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy"
Nov 03 13:23:26.241201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.241133    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333" containerName="revoke-keys"
Nov 03 13:23:26.241201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.241146    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" containerName="istio-proxy"
Nov 03 13:23:26.246084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.246057    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:26.251851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.251826    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"]
Nov 03 13:23:26.255520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.255498    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d3ca48\""
Nov 03 13:23:26.257261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.257024    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:26.260445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.260420    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"]
Nov 03 13:23:26.268248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.267862    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d3ca48-dockercfg-t5nf7\""
Nov 03 13:23:26.271079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.271058    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"]
Nov 03 13:23:26.329805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.329780    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs\" (UID: \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:26.329955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.329938    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwf4j\" (UniqueName: \"kubernetes.io/projected/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-kube-api-access-xwf4j\") pod \"payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4\" (UID: \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:26.330037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.330023    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htb7l\" (UniqueName: \"kubernetes.io/projected/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-kube-api-access-htb7l\") pod \"payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs\" (UID: \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:26.330080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.330054    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4\" (UID: \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:26.382918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.382891    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:26.431492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.431465    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwf4j\" (UniqueName: \"kubernetes.io/projected/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-kube-api-access-xwf4j\") pod \"payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4\" (UID: \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:26.431646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.431530    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htb7l\" (UniqueName: \"kubernetes.io/projected/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-kube-api-access-htb7l\") pod \"payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs\" (UID: \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:26.431646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.431561    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4\" (UID: \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:26.431779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.431672    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs\" (UID: \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:26.432376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.432357    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4\" (UID: \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:26.432572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.432553    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs\" (UID: \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:26.446535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.446508    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htb7l\" (UniqueName: \"kubernetes.io/projected/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-kube-api-access-htb7l\") pod \"payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs\" (UID: \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:26.446926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.446661    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwf4j\" (UniqueName: \"kubernetes.io/projected/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-kube-api-access-xwf4j\") pod \"payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4\" (UID: \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:26.528980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.528546    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:26.554476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.554440    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj" podUID="41b89cf0-3d7f-486b-bd21-fe8193a2ea89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:23:26.566584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.566553    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:26.579477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.579446    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:26.896256 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:26.896222    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22e671e_d5c5_4cc0_b357_74eddb4cbf7d.slice/crio-d75e7997331cc884b48749057d3f617fa524bcf34d7d39109380554f0c818203 WatchSource:0}: Error finding container d75e7997331cc884b48749057d3f617fa524bcf34d7d39109380554f0c818203: Status 404 returned error can't find the container with id d75e7997331cc884b48749057d3f617fa524bcf34d7d39109380554f0c818203
Nov 03 13:23:26.897052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.897029    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"]
Nov 03 13:23:26.931660 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:26.931635    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2bd68c_986a_4fdc_aafd_d398b8b7d87e.slice/crio-a09d88cbf93ecc81a73bc5d1f3ba253d95bf37e3a2b5922f258871951960246a WatchSource:0}: Error finding container a09d88cbf93ecc81a73bc5d1f3ba253d95bf37e3a2b5922f258871951960246a: Status 404 returned error can't find the container with id a09d88cbf93ecc81a73bc5d1f3ba253d95bf37e3a2b5922f258871951960246a
Nov 03 13:23:26.934926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:26.934899    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"]
Nov 03 13:23:27.085980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.085952    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2cb690a-f1c0-4871-8c1b-203584b7645e" path="/var/lib/kubelet/pods/d2cb690a-f1c0-4871-8c1b-203584b7645e/volumes"
Nov 03 13:23:27.194592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.194509    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:27.272885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.272827    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs" event={"ID":"af2bd68c-986a-4fdc-aafd-d398b8b7d87e","Type":"ContainerStarted","Data":"ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821"}
Nov 03 13:23:27.274423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.274386    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs" event={"ID":"af2bd68c-986a-4fdc-aafd-d398b8b7d87e","Type":"ContainerStarted","Data":"a09d88cbf93ecc81a73bc5d1f3ba253d95bf37e3a2b5922f258871951960246a"}
Nov 03 13:23:27.274625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.274432    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:27.277317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.277284    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4" event={"ID":"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d","Type":"ContainerStarted","Data":"41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee"}
Nov 03 13:23:27.277426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.277388    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:27.277503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.277435    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4" event={"ID":"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d","Type":"ContainerStarted","Data":"d75e7997331cc884b48749057d3f617fa524bcf34d7d39109380554f0c818203"}
Nov 03 13:23:27.294000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.293947    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs" podStartSLOduration=1.293933169 podStartE2EDuration="1.293933169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:27.290280762 +0000 UTC m=+848.368045788" watchObservedRunningTime="2026-11-03 13:23:27.293933169 +0000 UTC m=+848.371698199"
Nov 03 13:23:27.311030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.310984    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4" podStartSLOduration=1.310971479 podStartE2EDuration="1.310971479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:27.306244106 +0000 UTC m=+848.384009134" watchObservedRunningTime="2026-11-03 13:23:27.310971479 +0000 UTC m=+848.388736529"
Nov 03 13:23:27.883381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:27.883352    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:28.626234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:28.622639    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:30.230235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.226326    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Nov 03 13:23:30.256740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.256704    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:30.813238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.812963    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 13:23:30.818548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.817458    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"]
Nov 03 13:23:30.909351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.909300    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:30.943235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.927332    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:30.943235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.935547    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"]
Nov 03 13:23:30.943235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.935663    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:30.968234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.958552    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:23:30.968234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.963037    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:23:30.968234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.963092    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:23:30.968234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.963177    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:23:30.969575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:30.969542    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:31.070870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.070774    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71060761-vwjll\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.070870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.070841    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71060761-vwjll\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.071103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.070951    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-827dq\" (UniqueName: \"kubernetes.io/projected/7f6895e9-17e7-4555-90a9-4552dc4e2628-kube-api-access-827dq\") pod \"maas-api-revoke-sub-71060761-vwjll\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.172200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.172161    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71060761-vwjll\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.172384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.172318    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71060761-vwjll\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.172384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.172361    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-827dq\" (UniqueName: \"kubernetes.io/projected/7f6895e9-17e7-4555-90a9-4552dc4e2628-kube-api-access-827dq\") pod \"maas-api-revoke-sub-71060761-vwjll\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.172839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.172816    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71060761-vwjll\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.175105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.175078    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71060761-vwjll\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.181076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.181058    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-827dq\" (UniqueName: \"kubernetes.io/projected/7f6895e9-17e7-4555-90a9-4552dc4e2628-kube-api-access-827dq\") pod \"maas-api-revoke-sub-71060761-vwjll\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.186070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.186050    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:31.295774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.295743    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:31.341690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.341601    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:31.477294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.477253    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:31.592114 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:31.592049    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6895e9_17e7_4555_90a9_4552dc4e2628.slice/crio-4b43e9771832caf53c72e4a80520e3d97a3720b49c3f90a726376cac4f2c52ed WatchSource:0}: Error finding container 4b43e9771832caf53c72e4a80520e3d97a3720b49c3f90a726376cac4f2c52ed: Status 404 returned error can't find the container with id 4b43e9771832caf53c72e4a80520e3d97a3720b49c3f90a726376cac4f2c52ed
Nov 03 13:23:31.594345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.594320    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"]
Nov 03 13:23:31.991013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:31.990970    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:32.440803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:32.440758    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll" event={"ID":"7f6895e9-17e7-4555-90a9-4552dc4e2628","Type":"ContainerStarted","Data":"8b54b81a3f03bd101fcbe27c2824a07992d5fd7354c1dd0c429d163fcee4d236"}
Nov 03 13:23:32.441293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:32.440817    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll" event={"ID":"7f6895e9-17e7-4555-90a9-4552dc4e2628","Type":"ContainerStarted","Data":"4b43e9771832caf53c72e4a80520e3d97a3720b49c3f90a726376cac4f2c52ed"}
Nov 03 13:23:32.461874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:32.461829    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll" podStartSLOduration=2.461814713 podStartE2EDuration="2.461814713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:32.456513827 +0000 UTC m=+853.534278853" watchObservedRunningTime="2026-11-03 13:23:32.461814713 +0000 UTC m=+853.539579738"
Nov 03 13:23:32.899946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:32.899904    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:33.450176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:33.450142    5420 generic.go:356] "Generic (PLEG): container finished" podID="7f6895e9-17e7-4555-90a9-4552dc4e2628" containerID="8b54b81a3f03bd101fcbe27c2824a07992d5fd7354c1dd0c429d163fcee4d236" exitCode=0
Nov 03 13:23:33.450630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:33.450183    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll" event={"ID":"7f6895e9-17e7-4555-90a9-4552dc4e2628","Type":"ContainerDied","Data":"8b54b81a3f03bd101fcbe27c2824a07992d5fd7354c1dd0c429d163fcee4d236"}
Nov 03 13:23:33.628233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:33.627682    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:34.189694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.189658    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:34.620170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.620141    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:34.674347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.674315    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"]
Nov 03 13:23:34.675322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.675302    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7f6895e9-17e7-4555-90a9-4552dc4e2628" containerName="revoke-keys"
Nov 03 13:23:34.675401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.675326    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6895e9-17e7-4555-90a9-4552dc4e2628" containerName="revoke-keys"
Nov 03 13:23:34.675530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.675508    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6895e9-17e7-4555-90a9-4552dc4e2628" containerName="revoke-keys"
Nov 03 13:23:34.682868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.682850    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:34.689477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.689455    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"]
Nov 03 13:23:34.755546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.755474    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-openshift-ca\") pod \"7f6895e9-17e7-4555-90a9-4552dc4e2628\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") "
Nov 03 13:23:34.755693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.755645    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6895e9-17e7-4555-90a9-4552dc4e2628-kube-api-access-827dq\" (UniqueName: \"kubernetes.io/projected/7f6895e9-17e7-4555-90a9-4552dc4e2628-kube-api-access-827dq\") pod \"7f6895e9-17e7-4555-90a9-4552dc4e2628\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") "
Nov 03 13:23:34.755743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.755697    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-xks-ca\") pod \"7f6895e9-17e7-4555-90a9-4552dc4e2628\" (UID: \"7f6895e9-17e7-4555-90a9-4552dc4e2628\") "
Nov 03 13:23:34.755855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.755827    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-openshift-ca" pod "7f6895e9-17e7-4555-90a9-4552dc4e2628" (UID: "7f6895e9-17e7-4555-90a9-4552dc4e2628"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:34.756193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.756176    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:34.757641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.757619    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6895e9-17e7-4555-90a9-4552dc4e2628-kube-api-access-827dq" pod "7f6895e9-17e7-4555-90a9-4552dc4e2628" (UID: "7f6895e9-17e7-4555-90a9-4552dc4e2628"). InnerVolumeSpecName "kube-api-access-827dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:34.757722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.757689    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-xks-ca" pod "7f6895e9-17e7-4555-90a9-4552dc4e2628" (UID: "7f6895e9-17e7-4555-90a9-4552dc4e2628"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:34.856971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.856937    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7nq\" (UniqueName: \"kubernetes.io/projected/9c51a890-a003-4b94-9d1e-8517bbe774e5-kube-api-access-px7nq\") pod \"maas-api-revoke-sub-770053ed-z2ww9\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:34.857153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.856987    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-770053ed-z2ww9\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:34.857204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.857149    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-770053ed-z2ww9\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:34.857327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.857312    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-827dq\" (UniqueName: \"kubernetes.io/projected/7f6895e9-17e7-4555-90a9-4552dc4e2628-kube-api-access-827dq\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:34.857383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.857330    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6895e9-17e7-4555-90a9-4552dc4e2628-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:34.886023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.885992    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:34.958786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.958754    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-770053ed-z2ww9\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:34.958966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.958842    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-770053ed-z2ww9\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:34.959056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.959037    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7nq\" (UniqueName: \"kubernetes.io/projected/9c51a890-a003-4b94-9d1e-8517bbe774e5-kube-api-access-px7nq\") pod \"maas-api-revoke-sub-770053ed-z2ww9\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:34.959578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.959555    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-770053ed-z2ww9\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:34.961126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.961101    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-770053ed-z2ww9\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:34.967560 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:34.967540    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7nq\" (UniqueName: \"kubernetes.io/projected/9c51a890-a003-4b94-9d1e-8517bbe774e5-kube-api-access-px7nq\") pod \"maas-api-revoke-sub-770053ed-z2ww9\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:35.002454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:35.002427    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:35.282969 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:35.282944    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c51a890_a003_4b94_9d1e_8517bbe774e5.slice/crio-8ff7f8b4d46c82adb99af0b762f27564e348e08666e8ff6b952df74e20980c33 WatchSource:0}: Error finding container 8ff7f8b4d46c82adb99af0b762f27564e348e08666e8ff6b952df74e20980c33: Status 404 returned error can't find the container with id 8ff7f8b4d46c82adb99af0b762f27564e348e08666e8ff6b952df74e20980c33
Nov 03 13:23:35.284058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:35.284030    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"]
Nov 03 13:23:35.469003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:35.468971    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9" event={"ID":"9c51a890-a003-4b94-9d1e-8517bbe774e5","Type":"ContainerStarted","Data":"8ff7f8b4d46c82adb99af0b762f27564e348e08666e8ff6b952df74e20980c33"}
Nov 03 13:23:35.470466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:35.470443    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"
Nov 03 13:23:35.470604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:35.470488    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll" event={"ID":"7f6895e9-17e7-4555-90a9-4552dc4e2628","Type":"ContainerDied","Data":"4b43e9771832caf53c72e4a80520e3d97a3720b49c3f90a726376cac4f2c52ed"}
Nov 03 13:23:35.470604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:35.470540    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b43e9771832caf53c72e4a80520e3d97a3720b49c3f90a726376cac4f2c52ed"
Nov 03 13:23:35.493133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:35.493099    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:36.053041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.052981    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:23:36.178485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.178453    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-kube-api-access-v6mrv\" (UniqueName: \"kubernetes.io/projected/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-kube-api-access-v6mrv\") pod \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\" (UID: \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\") "
Nov 03 13:23:36.178714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.178659    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-plugins-config-volume\") pod \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\" (UID: \"41b89cf0-3d7f-486b-bd21-fe8193a2ea89\") "
Nov 03 13:23:36.179020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.178995    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-plugins-config-volume" pod "41b89cf0-3d7f-486b-bd21-fe8193a2ea89" (UID: "41b89cf0-3d7f-486b-bd21-fe8193a2ea89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:36.179251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.179232    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:36.180761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.180729    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-kube-api-access-v6mrv" pod "41b89cf0-3d7f-486b-bd21-fe8193a2ea89" (UID: "41b89cf0-3d7f-486b-bd21-fe8193a2ea89"). InnerVolumeSpecName "kube-api-access-v6mrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:36.280137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.280108    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6mrv\" (UniqueName: \"kubernetes.io/projected/41b89cf0-3d7f-486b-bd21-fe8193a2ea89-kube-api-access-v6mrv\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:36.346868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.346790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:36.480584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.480547    5420 generic.go:356] "Generic (PLEG): container finished" podID="41b89cf0-3d7f-486b-bd21-fe8193a2ea89" containerID="e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e" exitCode=0
Nov 03 13:23:36.480762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.480618    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"
Nov 03 13:23:36.480762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.480633    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj" event={"ID":"41b89cf0-3d7f-486b-bd21-fe8193a2ea89","Type":"ContainerDied","Data":"e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e"}
Nov 03 13:23:36.480762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.480688    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj" event={"ID":"41b89cf0-3d7f-486b-bd21-fe8193a2ea89","Type":"ContainerDied","Data":"fea2d92ed7a12dcd7e1e1151e1b7d06dcf0c884b6481cf1b21261cfd974ee2ec"}
Nov 03 13:23:36.480762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.480712    5420 scope.go:122] "RemoveContainer" containerID="e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e"
Nov 03 13:23:36.482752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.482723    5420 generic.go:356] "Generic (PLEG): container finished" podID="9c51a890-a003-4b94-9d1e-8517bbe774e5" containerID="2e7861a29b44a98c5a89e26f1c188fc21540fe23a534a7e268d09cca01eba67a" exitCode=0
Nov 03 13:23:36.482892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.482795    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9" event={"ID":"9c51a890-a003-4b94-9d1e-8517bbe774e5","Type":"ContainerDied","Data":"2e7861a29b44a98c5a89e26f1c188fc21540fe23a534a7e268d09cca01eba67a"}
Nov 03 13:23:36.493350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.493331    5420 scope.go:122] "RemoveContainer" containerID="e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e"
Nov 03 13:23:36.493607 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:23:36.493593    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e\": container with ID starting with e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e not found: ID does not exist" containerID="e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e"
Nov 03 13:23:36.493662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.493616    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e"} err="failed to get container status \"e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e\": rpc error: code = NotFound desc = could not find container \"e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e\": container with ID starting with e0a3076080bc6009f81a2767a6278f3a1a58bff08bb9f3f118a547242731be4e not found: ID does not exist"
Nov 03 13:23:36.525249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.525206    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"]
Nov 03 13:23:36.529917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.529893    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db71f4-57c44dc4f9-lqfgj"]
Nov 03 13:23:36.700558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:36.700509    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:37.085027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.084994    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b89cf0-3d7f-486b-bd21-fe8193a2ea89" path="/var/lib/kubelet/pods/41b89cf0-3d7f-486b-bd21-fe8193a2ea89/volumes"
Nov 03 13:23:37.324008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.323977    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:37.655779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.655758    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:37.696644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.696610    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-openshift-ca\") pod \"9c51a890-a003-4b94-9d1e-8517bbe774e5\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") "
Nov 03 13:23:37.696835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.696725    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c51a890-a003-4b94-9d1e-8517bbe774e5-kube-api-access-px7nq\" (UniqueName: \"kubernetes.io/projected/9c51a890-a003-4b94-9d1e-8517bbe774e5-kube-api-access-px7nq\") pod \"9c51a890-a003-4b94-9d1e-8517bbe774e5\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") "
Nov 03 13:23:37.696835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.696755    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-xks-ca\") pod \"9c51a890-a003-4b94-9d1e-8517bbe774e5\" (UID: \"9c51a890-a003-4b94-9d1e-8517bbe774e5\") "
Nov 03 13:23:37.696987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.696951    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-openshift-ca" pod "9c51a890-a003-4b94-9d1e-8517bbe774e5" (UID: "9c51a890-a003-4b94-9d1e-8517bbe774e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:37.697429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.697401    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:37.699087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.699026    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c51a890-a003-4b94-9d1e-8517bbe774e5-kube-api-access-px7nq" pod "9c51a890-a003-4b94-9d1e-8517bbe774e5" (UID: "9c51a890-a003-4b94-9d1e-8517bbe774e5"). InnerVolumeSpecName "kube-api-access-px7nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:37.699544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.699517    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-xks-ca" pod "9c51a890-a003-4b94-9d1e-8517bbe774e5" (UID: "9c51a890-a003-4b94-9d1e-8517bbe774e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:37.798846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.798764    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px7nq\" (UniqueName: \"kubernetes.io/projected/9c51a890-a003-4b94-9d1e-8517bbe774e5-kube-api-access-px7nq\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:37.798846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.798794    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51a890-a003-4b94-9d1e-8517bbe774e5-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:37.889562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:37.889503    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:38.318725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:38.318695    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:23:38.319188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:38.318762    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:23:38.504937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:38.504882    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9" event={"ID":"9c51a890-a003-4b94-9d1e-8517bbe774e5","Type":"ContainerDied","Data":"8ff7f8b4d46c82adb99af0b762f27564e348e08666e8ff6b952df74e20980c33"}
Nov 03 13:23:38.504937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:38.504904    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"
Nov 03 13:23:38.504937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:38.504941    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ff7f8b4d46c82adb99af0b762f27564e348e08666e8ff6b952df74e20980c33"
Nov 03 13:23:38.506254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:38.506015    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:39.083293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:39.083262    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:39.712757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:39.712718    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:40.378993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:40.378962    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:40.445787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:40.445753    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:23:41.062697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:41.062668    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:41.734913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:41.734874    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:43.193342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:43.193310    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:43.834976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:43.834908    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:44.505990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:44.505958    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:45.187902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.187870    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"]
Nov 03 13:23:45.188505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.188491    5420 cpu_manager.go:414] "containerMap: removing container" podUID="41b89cf0-3d7f-486b-bd21-fe8193a2ea89" containerName="payload-processing"
Nov 03 13:23:45.188564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.188506    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b89cf0-3d7f-486b-bd21-fe8193a2ea89" containerName="payload-processing"
Nov 03 13:23:45.188564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.188516    5420 cpu_manager.go:414] "containerMap: removing container" podUID="9c51a890-a003-4b94-9d1e-8517bbe774e5" containerName="revoke-keys"
Nov 03 13:23:45.188564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.188521    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c51a890-a003-4b94-9d1e-8517bbe774e5" containerName="revoke-keys"
Nov 03 13:23:45.188664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.188640    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b89cf0-3d7f-486b-bd21-fe8193a2ea89" containerName="payload-processing"
Nov 03 13:23:45.188664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.188648    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c51a890-a003-4b94-9d1e-8517bbe774e5" containerName="revoke-keys"
Nov 03 13:23:45.193759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.193734    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.197804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.197781    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:23:45.198678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.198657    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"]
Nov 03 13:23:45.199740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.199717    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:23:45.199740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.199723    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:23:45.199906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.199809    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:23:45.274846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.274811    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373412bc-e342-4779-acac-97e4fe59f367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd9cf826-wkgps\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.275020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.274980    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zld6s\" (UniqueName: \"kubernetes.io/projected/373412bc-e342-4779-acac-97e4fe59f367-kube-api-access-zld6s\") pod \"maas-api-revoke-sub-fd9cf826-wkgps\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.275087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.275071    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373412bc-e342-4779-acac-97e4fe59f367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd9cf826-wkgps\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.291749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.291716    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:45.376298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.376261    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373412bc-e342-4779-acac-97e4fe59f367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd9cf826-wkgps\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.376462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.376406    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373412bc-e342-4779-acac-97e4fe59f367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd9cf826-wkgps\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.376571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.376557    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zld6s\" (UniqueName: \"kubernetes.io/projected/373412bc-e342-4779-acac-97e4fe59f367-kube-api-access-zld6s\") pod \"maas-api-revoke-sub-fd9cf826-wkgps\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.377023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.376994    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373412bc-e342-4779-acac-97e4fe59f367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd9cf826-wkgps\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.379517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.379495    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373412bc-e342-4779-acac-97e4fe59f367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd9cf826-wkgps\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.388305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.388267    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zld6s\" (UniqueName: \"kubernetes.io/projected/373412bc-e342-4779-acac-97e4fe59f367-kube-api-access-zld6s\") pod \"maas-api-revoke-sub-fd9cf826-wkgps\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.512741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.512706    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:45.795511 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:45.795480    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373412bc_e342_4779_acac_97e4fe59f367.slice/crio-a70e20771a74f907f8b3f945f240cc1fd69d63e7c760a86395b2cc993dc871e7 WatchSource:0}: Error finding container a70e20771a74f907f8b3f945f240cc1fd69d63e7c760a86395b2cc993dc871e7: Status 404 returned error can't find the container with id a70e20771a74f907f8b3f945f240cc1fd69d63e7c760a86395b2cc993dc871e7
Nov 03 13:23:45.796822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.796798    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"]
Nov 03 13:23:45.830617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:45.830578    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:46.016204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.016173    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"]
Nov 03 13:23:46.021805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.021770    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:46.021942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.021815    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.029981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.029952    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"]
Nov 03 13:23:46.041482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.041452    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"]
Nov 03 13:23:46.058818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.058783    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7c2d03-hmgbt"]
Nov 03 13:23:46.088334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.088302    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40055367-g8k28\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.088517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.088496    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkd8\" (UniqueName: \"kubernetes.io/projected/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-kube-api-access-hnkd8\") pod \"maas-api-revoke-sub-40055367-g8k28\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.088568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.088532    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40055367-g8k28\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.190598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.190561    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkd8\" (UniqueName: \"kubernetes.io/projected/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-kube-api-access-hnkd8\") pod \"maas-api-revoke-sub-40055367-g8k28\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.190797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.190607    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40055367-g8k28\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.190797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.190789    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40055367-g8k28\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.191468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.191438    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40055367-g8k28\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.192950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.192930    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40055367-g8k28\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.198605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.198584    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkd8\" (UniqueName: \"kubernetes.io/projected/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-kube-api-access-hnkd8\") pod \"maas-api-revoke-sub-40055367-g8k28\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.352929 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.352852    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:46.406813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.406759    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:46.642623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.642528    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps" event={"ID":"373412bc-e342-4779-acac-97e4fe59f367","Type":"ContainerStarted","Data":"7ad95593e6051305f6551d1f7c03055b1a47aa5c396a82d4d7b20049fa1867b1"}
Nov 03 13:23:46.642993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.642681    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps" event={"ID":"373412bc-e342-4779-acac-97e4fe59f367","Type":"ContainerStarted","Data":"a70e20771a74f907f8b3f945f240cc1fd69d63e7c760a86395b2cc993dc871e7"}
Nov 03 13:23:46.646272 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:46.646243    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adc27ef_51fe_40fd_9375_d97ef8cef7f5.slice/crio-4402568a4cfcab32edcd5f164cc5e833160bd734fb31270be777c214b7558c9f WatchSource:0}: Error finding container 4402568a4cfcab32edcd5f164cc5e833160bd734fb31270be777c214b7558c9f: Status 404 returned error can't find the container with id 4402568a4cfcab32edcd5f164cc5e833160bd734fb31270be777c214b7558c9f
Nov 03 13:23:46.647818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.647784    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"]
Nov 03 13:23:46.678258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.678171    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps" podStartSLOduration=1.678151389 podStartE2EDuration="1.678151389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:46.673163187 +0000 UTC m=+867.750928214" watchObservedRunningTime="2026-11-03 13:23:46.678151389 +0000 UTC m=+867.755916416"
Nov 03 13:23:46.702698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.702665    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:46.851505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.851476    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:46.999320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:46.999292    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:47.161202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.159408    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6a2b36-49d6-42db-9f69-d51155955377" path="/var/lib/kubelet/pods/bc6a2b36-49d6-42db-9f69-d51155955377/volumes"
Nov 03 13:23:47.615872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.615839    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"]
Nov 03 13:23:47.620484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.620460    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.629846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.629821    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-79eca78c666a\""
Nov 03 13:23:47.636417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.636389    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"]
Nov 03 13:23:47.657787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.657750    5420 generic.go:356] "Generic (PLEG): container finished" podID="373412bc-e342-4779-acac-97e4fe59f367" containerID="7ad95593e6051305f6551d1f7c03055b1a47aa5c396a82d4d7b20049fa1867b1" exitCode=0
Nov 03 13:23:47.658316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.657888    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps" event={"ID":"373412bc-e342-4779-acac-97e4fe59f367","Type":"ContainerDied","Data":"7ad95593e6051305f6551d1f7c03055b1a47aa5c396a82d4d7b20049fa1867b1"}
Nov 03 13:23:47.660007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.659979    5420 generic.go:356] "Generic (PLEG): container finished" podID="9adc27ef-51fe-40fd-9375-d97ef8cef7f5" containerID="f91eb848b76e4964343a20744b87071871b52c818bd12af5b71089cca24b10be" exitCode=0
Nov 03 13:23:47.660127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.660057    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28" event={"ID":"9adc27ef-51fe-40fd-9375-d97ef8cef7f5","Type":"ContainerDied","Data":"f91eb848b76e4964343a20744b87071871b52c818bd12af5b71089cca24b10be"}
Nov 03 13:23:47.660127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.660098    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28" event={"ID":"9adc27ef-51fe-40fd-9375-d97ef8cef7f5","Type":"ContainerStarted","Data":"4402568a4cfcab32edcd5f164cc5e833160bd734fb31270be777c214b7558c9f"}
Nov 03 13:23:47.715121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.715083    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"]
Nov 03 13:23:47.715812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.715790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:47.726165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.726140    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.726882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.726856    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.727045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.726899    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.727112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.727050    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.727184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.727167    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6bxn\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-kube-api-access-d6bxn\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.729955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.729936    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"]
Nov 03 13:23:47.732275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.732256    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"simulator-w3-d3ca48-kserve-self-signed-certs\""
Nov 03 13:23:47.734348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.734327    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"default-dockercfg-wf2rg\""
Nov 03 13:23:47.775017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.774932    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"openshift-service-ca.crt\""
Nov 03 13:23:47.775144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.774996    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"kube-root-ca.crt\""
Nov 03 13:23:47.828376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828345    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdvk\" (UniqueName: \"kubernetes.io/projected/8506a60e-792e-4877-804a-d998bda6b1aa-kube-api-access-6mdvk\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.828537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828388    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-model-cache\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.828537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828471    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-tmp-dir\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.828609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828538    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.828644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828610    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6bxn\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-kube-api-access-d6bxn\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.828677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828644    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8506a60e-792e-4877-804a-d998bda6b1aa-tls-certs\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.828790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828773    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-home\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.828891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828871    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.828937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828906    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-dshm\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.828975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.828933    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.829428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.829409    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.829597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.829570    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.831552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.831532    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.841091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.841065    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6bxn\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-kube-api-access-d6bxn\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.925903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.925747    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:47.931318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.930013    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8506a60e-792e-4877-804a-d998bda6b1aa-tls-certs\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.931318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.930198    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-home\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.931318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.930291    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-dshm\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.931318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.930344    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdvk\" (UniqueName: \"kubernetes.io/projected/8506a60e-792e-4877-804a-d998bda6b1aa-kube-api-access-6mdvk\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.931318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.930385    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-model-cache\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.931318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.930431    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-tmp-dir\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.931318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.930936    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-tmp-dir\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.932189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.932160    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-model-cache\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.932473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.932441    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-home\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.936166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.935742    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8506a60e-792e-4877-804a-d998bda6b1aa-tls-certs\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.936166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.935962    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:47.938674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.938649    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-dshm\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:47.951917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:47.951683    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdvk\" (UniqueName: \"kubernetes.io/projected/8506a60e-792e-4877-804a-d998bda6b1aa-kube-api-access-6mdvk\") pod \"simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") " pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:48.046872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.046785    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:48.131095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.130987    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:48.247570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.247543    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"]
Nov 03 13:23:48.358035 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:23:48.358003    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8506a60e_792e_4877_804a_d998bda6b1aa.slice/crio-1b563e62008dc214bba97e229060d31787af95adea117d8b7d6e5cee4343bc7c WatchSource:0}: Error finding container 1b563e62008dc214bba97e229060d31787af95adea117d8b7d6e5cee4343bc7c: Status 404 returned error can't find the container with id 1b563e62008dc214bba97e229060d31787af95adea117d8b7d6e5cee4343bc7c
Nov 03 13:23:48.362289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.362265    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"]
Nov 03 13:23:48.475597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.475565    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:48.673667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.673564    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt" event={"ID":"8506a60e-792e-4877-804a-d998bda6b1aa","Type":"ContainerStarted","Data":"81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c"}
Nov 03 13:23:48.674159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.673936    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt" event={"ID":"8506a60e-792e-4877-804a-d998bda6b1aa","Type":"ContainerStarted","Data":"1b563e62008dc214bba97e229060d31787af95adea117d8b7d6e5cee4343bc7c"}
Nov 03 13:23:48.676133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.676104    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv" event={"ID":"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c","Type":"ContainerStarted","Data":"0c38d7316d71a1e4e60985030194c2e4ce61801e50028120be2b662a886520f1"}
Nov 03 13:23:48.676310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.676156    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv" event={"ID":"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c","Type":"ContainerStarted","Data":"b6227e298909e3b6cc4339c31816976e0502db38a4503b655d4df0e221b0d33a"}
Nov 03 13:23:48.695697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.695643    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt" podStartSLOduration=1.695627515 podStartE2EDuration="1.695627515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:48.692299552 +0000 UTC m=+869.770064589" watchObservedRunningTime="2026-11-03 13:23:48.695627515 +0000 UTC m=+869.773392541"
Nov 03 13:23:48.715195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.715133    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv" podStartSLOduration=1.7151133939999998 podStartE2EDuration="1.715113394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:48.71444714 +0000 UTC m=+869.792212167" watchObservedRunningTime="2026-11-03 13:23:48.715113394 +0000 UTC m=+869.792878421"
Nov 03 13:23:48.878584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.878555    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:48.930659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.929512    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:48.955462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.955176    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-openshift-ca\") pod \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") "
Nov 03 13:23:48.956429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.955786    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-kube-api-access-hnkd8\" (UniqueName: \"kubernetes.io/projected/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-kube-api-access-hnkd8\") pod \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") "
Nov 03 13:23:48.956429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.955941    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-xks-ca\") pod \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\" (UID: \"9adc27ef-51fe-40fd-9375-d97ef8cef7f5\") "
Nov 03 13:23:48.959981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.959941    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-openshift-ca" pod "9adc27ef-51fe-40fd-9375-d97ef8cef7f5" (UID: "9adc27ef-51fe-40fd-9375-d97ef8cef7f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:48.961932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.961883    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-xks-ca" pod "9adc27ef-51fe-40fd-9375-d97ef8cef7f5" (UID: "9adc27ef-51fe-40fd-9375-d97ef8cef7f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:48.964423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:48.964395    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-kube-api-access-hnkd8" pod "9adc27ef-51fe-40fd-9375-d97ef8cef7f5" (UID: "9adc27ef-51fe-40fd-9375-d97ef8cef7f5"). InnerVolumeSpecName "kube-api-access-hnkd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:49.048201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.048167    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:49.059311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.059282    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373412bc-e342-4779-acac-97e4fe59f367-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373412bc-e342-4779-acac-97e4fe59f367-maas-api-openshift-ca\") pod \"373412bc-e342-4779-acac-97e4fe59f367\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") "
Nov 03 13:23:49.059453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.059353    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373412bc-e342-4779-acac-97e4fe59f367-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373412bc-e342-4779-acac-97e4fe59f367-maas-api-xks-ca\") pod \"373412bc-e342-4779-acac-97e4fe59f367\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") "
Nov 03 13:23:49.059572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.059554    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373412bc-e342-4779-acac-97e4fe59f367-kube-api-access-zld6s\" (UniqueName: \"kubernetes.io/projected/373412bc-e342-4779-acac-97e4fe59f367-kube-api-access-zld6s\") pod \"373412bc-e342-4779-acac-97e4fe59f367\" (UID: \"373412bc-e342-4779-acac-97e4fe59f367\") "
Nov 03 13:23:49.059638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.059616    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373412bc-e342-4779-acac-97e4fe59f367-maas-api-openshift-ca" pod "373412bc-e342-4779-acac-97e4fe59f367" (UID: "373412bc-e342-4779-acac-97e4fe59f367"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:23:49.060105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.060085    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:49.060105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.060105    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373412bc-e342-4779-acac-97e4fe59f367-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:49.060292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.060115    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:49.060292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.060124    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkd8\" (UniqueName: \"kubernetes.io/projected/9adc27ef-51fe-40fd-9375-d97ef8cef7f5-kube-api-access-hnkd8\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:49.061371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.061355    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373412bc-e342-4779-acac-97e4fe59f367-maas-api-xks-ca" pod "373412bc-e342-4779-acac-97e4fe59f367" (UID: "373412bc-e342-4779-acac-97e4fe59f367"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:23:49.061514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.061495    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373412bc-e342-4779-acac-97e4fe59f367-kube-api-access-zld6s" pod "373412bc-e342-4779-acac-97e4fe59f367" (UID: "373412bc-e342-4779-acac-97e4fe59f367"). InnerVolumeSpecName "kube-api-access-zld6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:23:49.161955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.161923    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373412bc-e342-4779-acac-97e4fe59f367-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:49.161955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.161953    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zld6s\" (UniqueName: \"kubernetes.io/projected/373412bc-e342-4779-acac-97e4fe59f367-kube-api-access-zld6s\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:23:49.225798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.225755    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:49.685469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.685437    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"
Nov 03 13:23:49.685469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.685460    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps" event={"ID":"373412bc-e342-4779-acac-97e4fe59f367","Type":"ContainerDied","Data":"a70e20771a74f907f8b3f945f240cc1fd69d63e7c760a86395b2cc993dc871e7"}
Nov 03 13:23:49.685974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.685506    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a70e20771a74f907f8b3f945f240cc1fd69d63e7c760a86395b2cc993dc871e7"
Nov 03 13:23:49.687122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.687102    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"
Nov 03 13:23:49.687256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.687142    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28" event={"ID":"9adc27ef-51fe-40fd-9375-d97ef8cef7f5","Type":"ContainerDied","Data":"4402568a4cfcab32edcd5f164cc5e833160bd734fb31270be777c214b7558c9f"}
Nov 03 13:23:49.687256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.687186    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4402568a4cfcab32edcd5f164cc5e833160bd734fb31270be777c214b7558c9f"
Nov 03 13:23:49.687881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.687622    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:49.924118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:49.924031    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:50.642125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:50.642076    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:51.296046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:51.296017    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:51.899715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:51.899682    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:52.177886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:52.177804    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:52.686457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:52.686427    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:53.284651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:53.284621    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:53.896360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:53.896325    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:54.299352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:54.299324    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:54.481234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:54.480694    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:54.738644 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:23:54.738583    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:23:55.015797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:55.015713    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:55.233423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:55.233394    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"]
Nov 03 13:23:55.236495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:55.236466    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:23:55.722798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:55.722761    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:56.297188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:56.297142    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:57.009179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:57.009149    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:57.214708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:57.214639    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:57.379784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:57.379669    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:58.053233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:58.049195    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:58.053233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:58.051190    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:58.141016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:58.140563    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:23:59.234371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.234322    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Nov 03 13:23:59.286358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.286310    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:59.351351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.351302    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:59.387660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.387614    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:59.430358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.430306    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:59.680422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.680316    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:59.812840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.812800    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt" podUID="8506a60e-792e-4877-804a-d998bda6b1aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:59.821565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.820453    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:59.834856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.834812    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:59.852742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.852403    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Nov 03 13:23:59.864246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.860736    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:23:59.864246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:23:59.861446    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:00.235322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.235279    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv" podUID="2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" containerName="payload-processing" containerID="cri-o://0c38d7316d71a1e4e60985030194c2e4ce61801e50028120be2b662a886520f1" gracePeriod=25
Nov 03 13:24:00.236083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.236057    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Nov 03 13:24:00.238732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.236645    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:24:00.360264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.357595    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:24:00.459112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.459079    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:24:00.504096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.503993    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 13:24:00.524744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.522590    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:24:00.600087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.600037    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:00.606665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.606613    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:00.681641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.681597    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:00.682699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.682673    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:24:00.829401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.824124    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:24:00.829401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.824871    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:00.829401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.825457    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:24:00.854490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.854444    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:00.879245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.865300    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:00.880691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.880664    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:24:00.881252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.881234    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:24:00.934933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.934907    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:24:00.948145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.941120    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" containerID="0c38d7316d71a1e4e60985030194c2e4ce61801e50028120be2b662a886520f1" exitCode=0
Nov 03 13:24:00.948145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.942479    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv" event={"ID":"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c","Type":"ContainerDied","Data":"0c38d7316d71a1e4e60985030194c2e4ce61801e50028120be2b662a886520f1"}
Nov 03 13:24:00.954850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.953876    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:24:00.956627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.956584    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:24:00.962237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:00.961753    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:24:01.099828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.099751    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:24:01.112957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.112921    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:01.164347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.164317    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-routing-overlay-volume\") pod \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") "
Nov 03 13:24:01.164536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.164392    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-plugins-config-volume\") pod \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") "
Nov 03 13:24:01.164536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.164442    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-kube-api-access-d6bxn\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-kube-api-access-d6bxn\") pod \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") "
Nov 03 13:24:01.164757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.164725    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-routing-overlay-volume" pod "2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" (UID: "2c46e95a-c96a-4ca0-9a00-f138b05f3f1c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:01.164839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.164815    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-plugins-config-volume" pod "2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" (UID: "2c46e95a-c96a-4ca0-9a00-f138b05f3f1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:01.165133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.165116    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:01.165195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.165140    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:01.166639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.166612    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-kube-api-access-d6bxn" pod "2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" (UID: "2c46e95a-c96a-4ca0-9a00-f138b05f3f1c"). InnerVolumeSpecName "kube-api-access-d6bxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:01.265806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.265772    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-provider-credentials\") pod \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\" (UID: \"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c\") "
Nov 03 13:24:01.266237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.266153    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6bxn\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-kube-api-access-d6bxn\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:01.267887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.267855    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-provider-credentials" pod "2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" (UID: "2c46e95a-c96a-4ca0-9a00-f138b05f3f1c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:01.367557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.367178    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:01.485766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.485726    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"]
Nov 03 13:24:01.486504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.486487    5420 cpu_manager.go:414] "containerMap: removing container" podUID="9adc27ef-51fe-40fd-9375-d97ef8cef7f5" containerName="revoke-keys"
Nov 03 13:24:01.486588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.486507    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9adc27ef-51fe-40fd-9375-d97ef8cef7f5" containerName="revoke-keys"
Nov 03 13:24:01.486588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.486531    5420 cpu_manager.go:414] "containerMap: removing container" podUID="373412bc-e342-4779-acac-97e4fe59f367" containerName="revoke-keys"
Nov 03 13:24:01.486588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.486536    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373412bc-e342-4779-acac-97e4fe59f367" containerName="revoke-keys"
Nov 03 13:24:01.486588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.486544    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" containerName="payload-processing"
Nov 03 13:24:01.486588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.486550    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" containerName="payload-processing"
Nov 03 13:24:01.486745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.486670    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9adc27ef-51fe-40fd-9375-d97ef8cef7f5" containerName="revoke-keys"
Nov 03 13:24:01.486745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.486682    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="373412bc-e342-4779-acac-97e4fe59f367" containerName="revoke-keys"
Nov 03 13:24:01.486745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.486688    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" containerName="payload-processing"
Nov 03 13:24:01.490634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.490608    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.498725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.497052    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a2ccd8-openshift-default-dockercfg-sc96f\""
Nov 03 13:24:01.502177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.502156    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"]
Nov 03 13:24:01.568968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.568934    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.568968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.568971    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-data\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.569194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.569104    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-istio-token\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.569194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.569140    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b93da-47a4-45cf-a17d-836371568cfc-istiod-ca-cert\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.569194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.569171    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-certs\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.569362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.569342    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b93da-47a4-45cf-a17d-836371568cfc-istio-podinfo\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.569449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.569432    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-envoy\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.569501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.569484    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ln9\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-kube-api-access-j9ln9\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.569557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.569543    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-credential-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.670915    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.670953    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-data\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671055    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-istio-token\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671076    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b93da-47a4-45cf-a17d-836371568cfc-istiod-ca-cert\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671105    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-certs\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671149    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b93da-47a4-45cf-a17d-836371568cfc-istio-podinfo\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671252    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-envoy\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671296    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ln9\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-kube-api-access-j9ln9\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671347    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-credential-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671469    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671516    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-data\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671707    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-credential-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671772    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-certs\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.671944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.671921    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b93da-47a4-45cf-a17d-836371568cfc-istiod-ca-cert\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.673572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.673555    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-envoy\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.673787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.673767    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b93da-47a4-45cf-a17d-836371568cfc-istio-podinfo\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.681684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.681654    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-istio-token\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.683806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.683780    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ln9\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-kube-api-access-j9ln9\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.808447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.808412    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:01.856362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.856326    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:01.968172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.966763    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv" event={"ID":"2c46e95a-c96a-4ca0-9a00-f138b05f3f1c","Type":"ContainerDied","Data":"b6227e298909e3b6cc4339c31816976e0502db38a4503b655d4df0e221b0d33a"}
Nov 03 13:24:01.968172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.966881    5420 scope.go:122] "RemoveContainer" containerID="0c38d7316d71a1e4e60985030194c2e4ce61801e50028120be2b662a886520f1"
Nov 03 13:24:01.968172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:01.967101    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"
Nov 03 13:24:02.003129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:02.003097    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"]
Nov 03 13:24:02.008395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:02.008367    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-d5f6xn9cv"]
Nov 03 13:24:02.109002 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:02.108972    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31b93da_47a4_45cf_a17d_836371568cfc.slice/crio-0d9c3f85ba6dee479c098763c89c70d241dfca0d2695eacdf59192dbcf80fed0 WatchSource:0}: Error finding container 0d9c3f85ba6dee479c098763c89c70d241dfca0d2695eacdf59192dbcf80fed0: Status 404 returned error can't find the container with id 0d9c3f85ba6dee479c098763c89c70d241dfca0d2695eacdf59192dbcf80fed0
Nov 03 13:24:02.111691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:02.111659    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"]
Nov 03 13:24:02.111966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:02.111934    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:24:02.112054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:02.112004    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:24:02.112109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:02.112054    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:24:02.501830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:02.501797    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:02.979407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:02.979375    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" event={"ID":"d31b93da-47a4-45cf-a17d-836371568cfc","Type":"ContainerStarted","Data":"a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097"}
Nov 03 13:24:02.979605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:02.979427    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" event={"ID":"d31b93da-47a4-45cf-a17d-836371568cfc","Type":"ContainerStarted","Data":"0d9c3f85ba6dee479c098763c89c70d241dfca0d2695eacdf59192dbcf80fed0"}
Nov 03 13:24:03.006181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.006123    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" podStartSLOduration=2.006105232 podStartE2EDuration="2.006105232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:03.001054084 +0000 UTC m=+884.078819130" watchObservedRunningTime="2026-11-03 13:24:03.006105232 +0000 UTC m=+884.083870259"
Nov 03 13:24:03.086902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.086871    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c46e95a-c96a-4ca0-9a00-f138b05f3f1c" path="/var/lib/kubelet/pods/2c46e95a-c96a-4ca0-9a00-f138b05f3f1c/volumes"
Nov 03 13:24:03.189715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.189680    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:03.651696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.651659    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"]
Nov 03 13:24:03.657673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.657645    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.660527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.660496    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"]
Nov 03 13:24:03.664374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.664347    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:24:03.664520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.664386    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:24:03.666181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.666161    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:24:03.666384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.666181    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:24:03.702721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.702692    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvx7f\" (UniqueName: \"kubernetes.io/projected/1f596775-f524-4f4c-8ba9-473432865682-kube-api-access-bvx7f\") pod \"maas-api-revoke-sub-63a4c5a1-zxd2d\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.702897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.702873    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f596775-f524-4f4c-8ba9-473432865682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63a4c5a1-zxd2d\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.703023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.703007    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f596775-f524-4f4c-8ba9-473432865682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63a4c5a1-zxd2d\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.804796 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.804761    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f596775-f524-4f4c-8ba9-473432865682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63a4c5a1-zxd2d\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.804975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.804830    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvx7f\" (UniqueName: \"kubernetes.io/projected/1f596775-f524-4f4c-8ba9-473432865682-kube-api-access-bvx7f\") pod \"maas-api-revoke-sub-63a4c5a1-zxd2d\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.804975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.804930    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f596775-f524-4f4c-8ba9-473432865682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63a4c5a1-zxd2d\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.805561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.805542    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f596775-f524-4f4c-8ba9-473432865682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63a4c5a1-zxd2d\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.807177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.807154    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f596775-f524-4f4c-8ba9-473432865682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63a4c5a1-zxd2d\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.809199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.809177    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:03.811128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.811090    5420 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 13:24:03.811231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.811170    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 13:24:03.813599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.813581    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvx7f\" (UniqueName: \"kubernetes.io/projected/1f596775-f524-4f4c-8ba9-473432865682-kube-api-access-bvx7f\") pod \"maas-api-revoke-sub-63a4c5a1-zxd2d\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.978044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.978017    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:03.993724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:03.993689    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:04.341296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:04.340749    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"]
Nov 03 13:24:04.351572 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:04.351543    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f596775_f524_4f4c_8ba9_473432865682.slice/crio-c820692cb8cec4700443ef686d3e0543cf2f94a8dc8d0801b2c72448829aa91b WatchSource:0}: Error finding container c820692cb8cec4700443ef686d3e0543cf2f94a8dc8d0801b2c72448829aa91b: Status 404 returned error can't find the container with id c820692cb8cec4700443ef686d3e0543cf2f94a8dc8d0801b2c72448829aa91b
Nov 03 13:24:04.547293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:04.541115    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:04.711341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:04.711300    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:04.809543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:04.809504    5420 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 13:24:04.809734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:04.809564    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 13:24:05.005150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:05.005062    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:05.007163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:05.007132    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d" event={"ID":"1f596775-f524-4f4c-8ba9-473432865682","Type":"ContainerStarted","Data":"098d6a34d2e99d3cd4ca1b68062f9e5e4fe704aea82fc4dc80da04a78e335f2d"}
Nov 03 13:24:05.007330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:05.007183    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d" event={"ID":"1f596775-f524-4f4c-8ba9-473432865682","Type":"ContainerStarted","Data":"c820692cb8cec4700443ef686d3e0543cf2f94a8dc8d0801b2c72448829aa91b"}
Nov 03 13:24:05.036267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:05.028040    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d" podStartSLOduration=2.028020519 podStartE2EDuration="2.028020519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:05.025557339 +0000 UTC m=+886.103322365" watchObservedRunningTime="2026-11-03 13:24:05.028020519 +0000 UTC m=+886.105785546"
Nov 03 13:24:05.452991 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:05.452332    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:05.595325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:05.595295    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:05.762881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:05.762851    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:05.809591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:05.809557    5420 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 13:24:05.809781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:05.809614    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 13:24:06.016976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:06.016885    5420 generic.go:356] "Generic (PLEG): container finished" podID="1f596775-f524-4f4c-8ba9-473432865682" containerID="098d6a34d2e99d3cd4ca1b68062f9e5e4fe704aea82fc4dc80da04a78e335f2d" exitCode=0
Nov 03 13:24:06.017153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:06.016969    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d" event={"ID":"1f596775-f524-4f4c-8ba9-473432865682","Type":"ContainerDied","Data":"098d6a34d2e99d3cd4ca1b68062f9e5e4fe704aea82fc4dc80da04a78e335f2d"}
Nov 03 13:24:06.395407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:06.395319    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:06.579701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:06.577651    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:06.809147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:06.809113    5420 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 13:24:06.809581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:06.809175    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 13:24:07.257103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.257074    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:07.283118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.283075    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:07.399909 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.399818    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f596775-f524-4f4c-8ba9-473432865682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f596775-f524-4f4c-8ba9-473432865682-maas-api-xks-ca\") pod \"1f596775-f524-4f4c-8ba9-473432865682\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") "
Nov 03 13:24:07.400394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.399933    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f596775-f524-4f4c-8ba9-473432865682-kube-api-access-bvx7f\" (UniqueName: \"kubernetes.io/projected/1f596775-f524-4f4c-8ba9-473432865682-kube-api-access-bvx7f\") pod \"1f596775-f524-4f4c-8ba9-473432865682\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") "
Nov 03 13:24:07.400394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.400164    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f596775-f524-4f4c-8ba9-473432865682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f596775-f524-4f4c-8ba9-473432865682-maas-api-openshift-ca\") pod \"1f596775-f524-4f4c-8ba9-473432865682\" (UID: \"1f596775-f524-4f4c-8ba9-473432865682\") "
Nov 03 13:24:07.400874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.400845    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f596775-f524-4f4c-8ba9-473432865682-maas-api-openshift-ca" pod "1f596775-f524-4f4c-8ba9-473432865682" (UID: "1f596775-f524-4f4c-8ba9-473432865682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:07.402347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.402250    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f596775-f524-4f4c-8ba9-473432865682-maas-api-xks-ca" pod "1f596775-f524-4f4c-8ba9-473432865682" (UID: "1f596775-f524-4f4c-8ba9-473432865682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:07.402826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.402802    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f596775-f524-4f4c-8ba9-473432865682-kube-api-access-bvx7f" pod "1f596775-f524-4f4c-8ba9-473432865682" (UID: "1f596775-f524-4f4c-8ba9-473432865682"). InnerVolumeSpecName "kube-api-access-bvx7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:07.501814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.501780    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f596775-f524-4f4c-8ba9-473432865682-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:07.501814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.501812    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f596775-f524-4f4c-8ba9-473432865682-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:07.502012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.501828    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvx7f\" (UniqueName: \"kubernetes.io/projected/1f596775-f524-4f4c-8ba9-473432865682-kube-api-access-bvx7f\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:07.809161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.808943    5420 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 13:24:07.809161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.809024    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 13:24:07.940608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:07.940566    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:08.069687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.069606    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"
Nov 03 13:24:08.069687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.069643    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d" event={"ID":"1f596775-f524-4f4c-8ba9-473432865682","Type":"ContainerDied","Data":"c820692cb8cec4700443ef686d3e0543cf2f94a8dc8d0801b2c72448829aa91b"}
Nov 03 13:24:08.069872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.069700    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c820692cb8cec4700443ef686d3e0543cf2f94a8dc8d0801b2c72448829aa91b"
Nov 03 13:24:08.533457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.533419    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:08.552914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.552771    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"]
Nov 03 13:24:08.558147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.558112    5420 cpu_manager.go:414] "containerMap: removing container" podUID="1f596775-f524-4f4c-8ba9-473432865682" containerName="revoke-keys"
Nov 03 13:24:08.558392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.558372    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f596775-f524-4f4c-8ba9-473432865682" containerName="revoke-keys"
Nov 03 13:24:08.558754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.558730    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f596775-f524-4f4c-8ba9-473432865682" containerName="revoke-keys"
Nov 03 13:24:08.570531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.569249    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"]
Nov 03 13:24:08.570531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.569375    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.581241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.581071    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:24:08.582892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.582869    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:24:08.583043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.582994    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:24:08.583522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.583505    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:24:08.617354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.617320    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7228dc7-vxb6p\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.617677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.617651    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhd96\" (UniqueName: \"kubernetes.io/projected/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-kube-api-access-hhd96\") pod \"maas-api-revoke-sub-c7228dc7-vxb6p\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.617895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.617872    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7228dc7-vxb6p\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.718838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.718805    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7228dc7-vxb6p\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.719010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.718899    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhd96\" (UniqueName: \"kubernetes.io/projected/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-kube-api-access-hhd96\") pod \"maas-api-revoke-sub-c7228dc7-vxb6p\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.719010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.718923    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7228dc7-vxb6p\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.719737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.719711    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7228dc7-vxb6p\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.721456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.721431    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7228dc7-vxb6p\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.727757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.727732    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhd96\" (UniqueName: \"kubernetes.io/projected/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-kube-api-access-hhd96\") pod \"maas-api-revoke-sub-c7228dc7-vxb6p\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:08.813638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.813567    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:08.814085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.814064    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:08.814711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.814691    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:24:08.911150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:08.911121    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:09.202340 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:09.202306    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bc80b9_15ad_4c81_b1fe_48a61b4d9315.slice/crio-0f80175926ec75f064060df38289102aff86b23792cd0a4dc8e57242fc47549a WatchSource:0}: Error finding container 0f80175926ec75f064060df38289102aff86b23792cd0a4dc8e57242fc47549a: Status 404 returned error can't find the container with id 0f80175926ec75f064060df38289102aff86b23792cd0a4dc8e57242fc47549a
Nov 03 13:24:09.203770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.203747    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"]
Nov 03 13:24:09.283132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.283098    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:09.855399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.855325    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:24:09.876662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.876266    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"]
Nov 03 13:24:09.876662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.876551    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy" containerID="cri-o://11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc" gracePeriod=30
Nov 03 13:24:09.877703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.877675    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:09.883412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.883381    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"]
Nov 03 13:24:09.883664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.883636    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy" containerID="cri-o://7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce" gracePeriod=30
Nov 03 13:24:09.892026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.891997    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"]
Nov 03 13:24:09.892232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:09.892195    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy" containerID="cri-o://34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03" gracePeriod=30
Nov 03 13:24:10.088125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:10.088075    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p" event={"ID":"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315","Type":"ContainerStarted","Data":"c95a13d94f4e8678dcfafd29068fd1c84b82da2b1dfb1e89aa770e7d577a5bdb"}
Nov 03 13:24:10.088465 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:10.088143    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p" event={"ID":"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315","Type":"ContainerStarted","Data":"0f80175926ec75f064060df38289102aff86b23792cd0a4dc8e57242fc47549a"}
Nov 03 13:24:10.109834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:10.109746    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p" podStartSLOduration=2.109733263 podStartE2EDuration="2.109733263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:10.106103877 +0000 UTC m=+891.183868904" watchObservedRunningTime="2026-11-03 13:24:10.109733263 +0000 UTC m=+891.187498327"
Nov 03 13:24:10.655848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:10.655808    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:11.150356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:11.150311    5420 generic.go:356] "Generic (PLEG): container finished" podID="c4bc80b9-15ad-4c81-b1fe-48a61b4d9315" containerID="c95a13d94f4e8678dcfafd29068fd1c84b82da2b1dfb1e89aa770e7d577a5bdb" exitCode=0
Nov 03 13:24:11.199350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:11.195400    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p" event={"ID":"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315","Type":"ContainerDied","Data":"c95a13d94f4e8678dcfafd29068fd1c84b82da2b1dfb1e89aa770e7d577a5bdb"}
Nov 03 13:24:11.558361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:11.558329    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:11.980243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:11.975375    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6sg42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:24:11.980243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:11.975467    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:24:12.491235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:12.486254    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:12.897523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:12.889321    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc" podUID="0ee6c876-9bf8-4b10-89f8-60da53ccdfc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Nov 03 13:24:12.899711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:12.897772    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:12.899711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:12.898114    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt" podUID="8506a60e-792e-4877-804a-d998bda6b1aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.119:8000/ready\": context deadline exceeded"
Nov 03 13:24:12.948173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:12.940657    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:13.010985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.010678    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:13.054246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.054116    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:13.310430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.310402    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:13.395232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.395083    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-openshift-ca\") pod \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") "
Nov 03 13:24:13.395232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.395149    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-kube-api-access-hhd96\" (UniqueName: \"kubernetes.io/projected/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-kube-api-access-hhd96\") pod \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") "
Nov 03 13:24:13.395482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.395362    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-xks-ca\") pod \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\" (UID: \"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315\") "
Nov 03 13:24:13.421635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.403086    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-openshift-ca" pod "c4bc80b9-15ad-4c81-b1fe-48a61b4d9315" (UID: "c4bc80b9-15ad-4c81-b1fe-48a61b4d9315"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:13.421635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.411713    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:13.480138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.479673    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p" event={"ID":"c4bc80b9-15ad-4c81-b1fe-48a61b4d9315","Type":"ContainerDied","Data":"0f80175926ec75f064060df38289102aff86b23792cd0a4dc8e57242fc47549a"}
Nov 03 13:24:13.480138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.479741    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f80175926ec75f064060df38289102aff86b23792cd0a4dc8e57242fc47549a"
Nov 03 13:24:13.480138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.479829    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"
Nov 03 13:24:13.522821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.512851    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:13.547504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.546357    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:13.551650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.550784    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-xks-ca" pod "c4bc80b9-15ad-4c81-b1fe-48a61b4d9315" (UID: "c4bc80b9-15ad-4c81-b1fe-48a61b4d9315"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:13.553126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.552915    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:13.610342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.556479    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:24:13.612163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.610650    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-kube-api-access-hhd96" pod "c4bc80b9-15ad-4c81-b1fe-48a61b4d9315" (UID: "c4bc80b9-15ad-4c81-b1fe-48a61b4d9315"). InnerVolumeSpecName "kube-api-access-hhd96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:13.653464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.644395    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:13.653464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.644429    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhd96\" (UniqueName: \"kubernetes.io/projected/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315-kube-api-access-hhd96\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:13.734112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.734000    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:13.860729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.860641    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:24:13.893057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.891394    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"]
Nov 03 13:24:13.893057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.892841    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c4bc80b9-15ad-4c81-b1fe-48a61b4d9315" containerName="revoke-keys"
Nov 03 13:24:13.893057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.892903    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bc80b9-15ad-4c81-b1fe-48a61b4d9315" containerName="revoke-keys"
Nov 03 13:24:13.893602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.893223    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bc80b9-15ad-4c81-b1fe-48a61b4d9315" containerName="revoke-keys"
Nov 03 13:24:13.956464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.956426    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"]
Nov 03 13:24:13.956650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.956587    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:13.959834 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:24:13.959803    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bc80b9_15ad_4c81_b1fe_48a61b4d9315.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 13:24:13.969226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.966072    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:24:13.969226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.966410    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:24:13.976431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.976127    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:24:13.976431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:13.976150    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:24:14.058421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.058387    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94kks\" (UniqueName: \"kubernetes.io/projected/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-kube-api-access-94kks\") pod \"maas-api-revoke-sub-b029f379-tnxlc\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.058598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.058507    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b029f379-tnxlc\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.058598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.058576    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b029f379-tnxlc\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.160125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.160007    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b029f379-tnxlc\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.160125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.160115    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b029f379-tnxlc\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.160388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.160349    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94kks\" (UniqueName: \"kubernetes.io/projected/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-kube-api-access-94kks\") pod \"maas-api-revoke-sub-b029f379-tnxlc\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.161097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.161075    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b029f379-tnxlc\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.162551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.162530    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b029f379-tnxlc\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.169467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.169448    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94kks\" (UniqueName: \"kubernetes.io/projected/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-kube-api-access-94kks\") pod \"maas-api-revoke-sub-b029f379-tnxlc\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.297631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.297598    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:14.510929 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.510896    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:14.588287 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:14.588250    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda040a8ac_bd8c_4078_b04d_c398e1f6ba9b.slice/crio-b1916e6adcdc5cb16046250ba5f630dec3d579b70672d05156f2e2d46843cc16 WatchSource:0}: Error finding container b1916e6adcdc5cb16046250ba5f630dec3d579b70672d05156f2e2d46843cc16: Status 404 returned error can't find the container with id b1916e6adcdc5cb16046250ba5f630dec3d579b70672d05156f2e2d46843cc16
Nov 03 13:24:14.589760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:14.589734    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"]
Nov 03 13:24:15.120228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.120142    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:15.158159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.157992    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:24:15.177263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.176968    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-istio-token\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-istio-token\") pod \"8968f39e-7bbd-4d21-afa6-67116c521656\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") "
Nov 03 13:24:15.177263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.177017    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-data\") pod \"8968f39e-7bbd-4d21-afa6-67116c521656\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") "
Nov 03 13:24:15.177263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.177064    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-certs\") pod \"8968f39e-7bbd-4d21-afa6-67116c521656\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") "
Nov 03 13:24:15.177263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.177093    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8968f39e-7bbd-4d21-afa6-67116c521656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8968f39e-7bbd-4d21-afa6-67116c521656-istiod-ca-cert\") pod \"8968f39e-7bbd-4d21-afa6-67116c521656\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") "
Nov 03 13:24:15.177263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.177146    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-socket\") pod \"8968f39e-7bbd-4d21-afa6-67116c521656\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") "
Nov 03 13:24:15.177263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.177225    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-credential-socket\") pod \"8968f39e-7bbd-4d21-afa6-67116c521656\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") "
Nov 03 13:24:15.177711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.177295    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-envoy\") pod \"8968f39e-7bbd-4d21-afa6-67116c521656\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") "
Nov 03 13:24:15.177711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.177404    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-kube-api-access-sbg75\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-kube-api-access-sbg75\") pod \"8968f39e-7bbd-4d21-afa6-67116c521656\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") "
Nov 03 13:24:15.177711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.177479    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8968f39e-7bbd-4d21-afa6-67116c521656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8968f39e-7bbd-4d21-afa6-67116c521656-istio-podinfo\") pod \"8968f39e-7bbd-4d21-afa6-67116c521656\" (UID: \"8968f39e-7bbd-4d21-afa6-67116c521656\") "
Nov 03 13:24:15.179334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.178786    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-socket" pod "8968f39e-7bbd-4d21-afa6-67116c521656" (UID: "8968f39e-7bbd-4d21-afa6-67116c521656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.179334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.178807    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-data" pod "8968f39e-7bbd-4d21-afa6-67116c521656" (UID: "8968f39e-7bbd-4d21-afa6-67116c521656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.179334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.178731    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-certs" pod "8968f39e-7bbd-4d21-afa6-67116c521656" (UID: "8968f39e-7bbd-4d21-afa6-67116c521656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.179334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.179087    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8968f39e-7bbd-4d21-afa6-67116c521656-istiod-ca-cert" pod "8968f39e-7bbd-4d21-afa6-67116c521656" (UID: "8968f39e-7bbd-4d21-afa6-67116c521656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:15.179640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.179391    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-credential-socket" pod "8968f39e-7bbd-4d21-afa6-67116c521656" (UID: "8968f39e-7bbd-4d21-afa6-67116c521656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.180540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.180323    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.180540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.180344    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.180540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.180357    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8968f39e-7bbd-4d21-afa6-67116c521656-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.180540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.180370    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.180540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.180384    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.189287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.187508    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-istio-token" pod "8968f39e-7bbd-4d21-afa6-67116c521656" (UID: "8968f39e-7bbd-4d21-afa6-67116c521656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:15.189287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.188472    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8968f39e-7bbd-4d21-afa6-67116c521656-istio-podinfo" pod "8968f39e-7bbd-4d21-afa6-67116c521656" (UID: "8968f39e-7bbd-4d21-afa6-67116c521656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:24:15.189287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.188542    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-kube-api-access-sbg75" pod "8968f39e-7bbd-4d21-afa6-67116c521656" (UID: "8968f39e-7bbd-4d21-afa6-67116c521656"). InnerVolumeSpecName "kube-api-access-sbg75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:15.190619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.190593    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-envoy" pod "8968f39e-7bbd-4d21-afa6-67116c521656" (UID: "8968f39e-7bbd-4d21-afa6-67116c521656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.262699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.262674    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.281036    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-certs\") pod \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") "
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.281187    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-socket\") pod \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") "
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.281632    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-certs" pod "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" (UID: "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.281945    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-socket" pod "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" (UID: "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.282450    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.281280    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istiod-ca-cert\") pod \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") "
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.282662    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-envoy\") pod \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") "
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.282722    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-kube-api-access-wvhww\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-kube-api-access-wvhww\") pod \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") "
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.282849    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-token\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-token\") pod \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") "
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.282876    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-credential-socket\") pod \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") "
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.282919    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-podinfo\") pod \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") "
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.282948    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-data\") pod \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\" (UID: \"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27\") "
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.283130    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istiod-ca-cert" pod "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" (UID: "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.283457    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-data" pod "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" (UID: "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.283852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.283765    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-credential-socket" pod "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" (UID: "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.284797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.284368    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.284797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.284390    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.284797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.284406    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.284797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.284421    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbg75\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-kube-api-access-sbg75\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.284797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.284434    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.284797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.284446    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8968f39e-7bbd-4d21-afa6-67116c521656-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.284797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.284458    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8968f39e-7bbd-4d21-afa6-67116c521656-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.284797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.284470    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.284797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.284483    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8968f39e-7bbd-4d21-afa6-67116c521656-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.286873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.286849    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-envoy" pod "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" (UID: "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.286997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.286902    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-token" pod "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" (UID: "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:15.287243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.287013    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-podinfo" pod "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" (UID: "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:24:15.288576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.288553    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-kube-api-access-wvhww" pod "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" (UID: "c29ba0ec-171e-44ef-96e0-e8fbd2a73e27"). InnerVolumeSpecName "kube-api-access-wvhww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:15.385916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.385826    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-envoy\") pod \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") "
Nov 03 13:24:15.386097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.385919    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-token\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-token\") pod \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") "
Nov 03 13:24:15.386097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.385949    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-podinfo\") pod \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") "
Nov 03 13:24:15.386097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.385979    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istiod-ca-cert\") pod \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") "
Nov 03 13:24:15.386097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.386084    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-credential-socket\") pod \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") "
Nov 03 13:24:15.386483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.386178    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-socket\") pod \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") "
Nov 03 13:24:15.386581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.386516    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istiod-ca-cert" pod "6f1cd1f4-1249-4347-b5bf-f75d94de573e" (UID: "6f1cd1f4-1249-4347-b5bf-f75d94de573e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:15.386581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.386565    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-socket" pod "6f1cd1f4-1249-4347-b5bf-f75d94de573e" (UID: "6f1cd1f4-1249-4347-b5bf-f75d94de573e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.386741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.386634    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-certs\") pod \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") "
Nov 03 13:24:15.386741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.386643    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-credential-socket" pod "6f1cd1f4-1249-4347-b5bf-f75d94de573e" (UID: "6f1cd1f4-1249-4347-b5bf-f75d94de573e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.386741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.386699    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-kube-api-access-9b627\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-kube-api-access-9b627\") pod \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") "
Nov 03 13:24:15.386982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.386845    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-data\") pod \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\" (UID: \"6f1cd1f4-1249-4347-b5bf-f75d94de573e\") "
Nov 03 13:24:15.387583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.387488    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.387583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.387516    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.387583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.387532    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.387583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.387547    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.387583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.387562    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.387583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.387576    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.388235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.387591    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhww\" (UniqueName: \"kubernetes.io/projected/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27-kube-api-access-wvhww\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.388235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.387673    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-data" pod "6f1cd1f4-1249-4347-b5bf-f75d94de573e" (UID: "6f1cd1f4-1249-4347-b5bf-f75d94de573e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.388235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.387803    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-certs" pod "6f1cd1f4-1249-4347-b5bf-f75d94de573e" (UID: "6f1cd1f4-1249-4347-b5bf-f75d94de573e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.388739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.388700    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-podinfo" pod "6f1cd1f4-1249-4347-b5bf-f75d94de573e" (UID: "6f1cd1f4-1249-4347-b5bf-f75d94de573e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:24:15.388846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.388822    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-token" pod "6f1cd1f4-1249-4347-b5bf-f75d94de573e" (UID: "6f1cd1f4-1249-4347-b5bf-f75d94de573e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:15.388925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.388910    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-envoy" pod "6f1cd1f4-1249-4347-b5bf-f75d94de573e" (UID: "6f1cd1f4-1249-4347-b5bf-f75d94de573e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:24:15.389916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.389896    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-kube-api-access-9b627" pod "6f1cd1f4-1249-4347-b5bf-f75d94de573e" (UID: "6f1cd1f4-1249-4347-b5bf-f75d94de573e"). InnerVolumeSpecName "kube-api-access-9b627". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:15.488755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.488721    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.488755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.488751    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.488755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.488760    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.488994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.488773    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1cd1f4-1249-4347-b5bf-f75d94de573e-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.488994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.488786    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1cd1f4-1249-4347-b5bf-f75d94de573e-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.488994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.488795    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b627\" (UniqueName: \"kubernetes.io/projected/6f1cd1f4-1249-4347-b5bf-f75d94de573e-kube-api-access-9b627\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:15.498484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.498446    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc" event={"ID":"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b","Type":"ContainerStarted","Data":"dcd14727bd19b475cbc7662ce0be2c108261e2d9b2d2ec2fa87d114cc49afdd8"}
Nov 03 13:24:15.498671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.498509    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc" event={"ID":"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b","Type":"ContainerStarted","Data":"b1916e6adcdc5cb16046250ba5f630dec3d579b70672d05156f2e2d46843cc16"}
Nov 03 13:24:15.499952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.499927    5420 generic.go:356] "Generic (PLEG): container finished" podID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerID="7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce" exitCode=0
Nov 03 13:24:15.500149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.499968    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"
Nov 03 13:24:15.500149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.500012    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" event={"ID":"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27","Type":"ContainerDied","Data":"7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce"}
Nov 03 13:24:15.500149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.500058    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr" event={"ID":"c29ba0ec-171e-44ef-96e0-e8fbd2a73e27","Type":"ContainerDied","Data":"b3b4c43ae2cc3baef1d582d37434a42ab7b9a22d92d1adcc521d1a2b75624b3b"}
Nov 03 13:24:15.500149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.500081    5420 scope.go:122] "RemoveContainer" containerID="7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce"
Nov 03 13:24:15.502408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.502380    5420 generic.go:356] "Generic (PLEG): container finished" podID="8968f39e-7bbd-4d21-afa6-67116c521656" containerID="34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03" exitCode=0
Nov 03 13:24:15.502542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.502420    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"
Nov 03 13:24:15.502542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.502466    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" event={"ID":"8968f39e-7bbd-4d21-afa6-67116c521656","Type":"ContainerDied","Data":"34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03"}
Nov 03 13:24:15.502542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.502532    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42" event={"ID":"8968f39e-7bbd-4d21-afa6-67116c521656","Type":"ContainerDied","Data":"f260255f8b880f748e9e877a3787aef8a83b896a85c6f2188740b24dfe37c963"}
Nov 03 13:24:15.504282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.504256    5420 generic.go:356] "Generic (PLEG): container finished" podID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerID="11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc" exitCode=0
Nov 03 13:24:15.504405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.504307    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"
Nov 03 13:24:15.504405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.504307    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" event={"ID":"6f1cd1f4-1249-4347-b5bf-f75d94de573e","Type":"ContainerDied","Data":"11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc"}
Nov 03 13:24:15.504522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.504441    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4" event={"ID":"6f1cd1f4-1249-4347-b5bf-f75d94de573e","Type":"ContainerDied","Data":"7b2ea0559f51dea24772874c5d6cf2a0294bcc7b0db434b5aded9a8bca33cf0a"}
Nov 03 13:24:15.512784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.512762    5420 scope.go:122] "RemoveContainer" containerID="7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce"
Nov 03 13:24:15.513127 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:24:15.513106    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce\": container with ID starting with 7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce not found: ID does not exist" containerID="7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce"
Nov 03 13:24:15.513192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.513140    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce"} err="failed to get container status \"7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce\": rpc error: code = NotFound desc = could not find container \"7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce\": container with ID starting with 7279f93d573f415bf2099990d8da1f9da9db84b865b38fe610600f19ec3dadce not found: ID does not exist"
Nov 03 13:24:15.513192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.513164    5420 scope.go:122] "RemoveContainer" containerID="34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03"
Nov 03 13:24:15.520705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.520666    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc" podStartSLOduration=2.520652385 podStartE2EDuration="2.520652385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:15.518687605 +0000 UTC m=+896.596452632" watchObservedRunningTime="2026-11-03 13:24:15.520652385 +0000 UTC m=+896.598417407"
Nov 03 13:24:15.524308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.524291    5420 scope.go:122] "RemoveContainer" containerID="8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738"
Nov 03 13:24:15.538352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.538329    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"]
Nov 03 13:24:15.543662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.543640    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ttvvr"]
Nov 03 13:24:15.554410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.554385    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"]
Nov 03 13:24:15.564563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.564541    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xz4c4"]
Nov 03 13:24:15.579721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.579685    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"]
Nov 03 13:24:15.584959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.584937    5420 scope.go:122] "RemoveContainer" containerID="34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03"
Nov 03 13:24:15.585426 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:24:15.585399    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03\": container with ID starting with 34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03 not found: ID does not exist" containerID="34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03"
Nov 03 13:24:15.585519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.585436    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03"} err="failed to get container status \"34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03\": rpc error: code = NotFound desc = could not find container \"34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03\": container with ID starting with 34fdfd4efee9a0a5d8fe7a44d210a8bfa6b4623fb392554dd554ae4df1956d03 not found: ID does not exist"
Nov 03 13:24:15.585519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.585461    5420 scope.go:122] "RemoveContainer" containerID="8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738"
Nov 03 13:24:15.585803 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:24:15.585781    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738\": container with ID starting with 8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738 not found: ID does not exist" containerID="8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738"
Nov 03 13:24:15.585856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.585810    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738"} err="failed to get container status \"8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738\": rpc error: code = NotFound desc = could not find container \"8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738\": container with ID starting with 8fb13f49045abfb8926f46d1a00077e710ae38c6b6bac83d2ecdc1f627302738 not found: ID does not exist"
Nov 03 13:24:15.585856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.585828    5420 scope.go:122] "RemoveContainer" containerID="11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc"
Nov 03 13:24:15.595968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.595941    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6sg42"]
Nov 03 13:24:15.598911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.598897    5420 scope.go:122] "RemoveContainer" containerID="11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc"
Nov 03 13:24:15.599248 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:24:15.599205    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc\": container with ID starting with 11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc not found: ID does not exist" containerID="11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc"
Nov 03 13:24:15.599344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.599258    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc"} err="failed to get container status \"11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc\": rpc error: code = NotFound desc = could not find container \"11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc\": container with ID starting with 11a801d18cc6ef3f116c2a7d1def815ed10359b47b1c902c534f98ecf7fa47fc not found: ID does not exist"
Nov 03 13:24:15.903912 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:15.903868    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:16.501321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:16.501289    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:16.531420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:16.531386    5420 generic.go:356] "Generic (PLEG): container finished" podID="a040a8ac-bd8c-4078-b04d-c398e1f6ba9b" containerID="dcd14727bd19b475cbc7662ce0be2c108261e2d9b2d2ec2fa87d114cc49afdd8" exitCode=0
Nov 03 13:24:16.531663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:16.531435    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc" event={"ID":"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b","Type":"ContainerDied","Data":"dcd14727bd19b475cbc7662ce0be2c108261e2d9b2d2ec2fa87d114cc49afdd8"}
Nov 03 13:24:17.085621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.085585    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" path="/var/lib/kubelet/pods/6f1cd1f4-1249-4347-b5bf-f75d94de573e/volumes"
Nov 03 13:24:17.086284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.086264    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" path="/var/lib/kubelet/pods/8968f39e-7bbd-4d21-afa6-67116c521656/volumes"
Nov 03 13:24:17.086782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.086764    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" path="/var/lib/kubelet/pods/c29ba0ec-171e-44ef-96e0-e8fbd2a73e27/volumes"
Nov 03 13:24:17.213296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.213253    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:17.366760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.366630    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:17.706235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.706198    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:17.790386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.790348    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:17.827754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.827715    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-kube-api-access-94kks\" (UniqueName: \"kubernetes.io/projected/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-kube-api-access-94kks\") pod \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") "
Nov 03 13:24:17.827945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.827830    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-xks-ca\") pod \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") "
Nov 03 13:24:17.827945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.827877    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-openshift-ca\") pod \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\" (UID: \"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b\") "
Nov 03 13:24:17.828377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.828345    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-openshift-ca" pod "a040a8ac-bd8c-4078-b04d-c398e1f6ba9b" (UID: "a040a8ac-bd8c-4078-b04d-c398e1f6ba9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:17.828759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.828737    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:17.830141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.830113    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-xks-ca" pod "a040a8ac-bd8c-4078-b04d-c398e1f6ba9b" (UID: "a040a8ac-bd8c-4078-b04d-c398e1f6ba9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:17.830340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.830296    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-kube-api-access-94kks" pod "a040a8ac-bd8c-4078-b04d-c398e1f6ba9b" (UID: "a040a8ac-bd8c-4078-b04d-c398e1f6ba9b"). InnerVolumeSpecName "kube-api-access-94kks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:17.929998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.929911    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94kks\" (UniqueName: \"kubernetes.io/projected/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-kube-api-access-94kks\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:17.929998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:17.929945    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:18.386734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:18.386705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:18.551579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:18.551552    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"
Nov 03 13:24:18.551749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:18.551592    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc" event={"ID":"a040a8ac-bd8c-4078-b04d-c398e1f6ba9b","Type":"ContainerDied","Data":"b1916e6adcdc5cb16046250ba5f630dec3d579b70672d05156f2e2d46843cc16"}
Nov 03 13:24:18.551749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:18.551640    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1916e6adcdc5cb16046250ba5f630dec3d579b70672d05156f2e2d46843cc16"
Nov 03 13:24:19.059388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:19.059359    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:19.222887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:19.222859    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:24:19.224658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:19.224637    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:24:19.699665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:19.699633    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:19.948028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:19.947992    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"]
Nov 03 13:24:20.456819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.456789    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"]
Nov 03 13:24:20.457433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457418    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy"
Nov 03 13:24:20.457487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457434    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy"
Nov 03 13:24:20.457487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457452    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy"
Nov 03 13:24:20.457487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457458    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy"
Nov 03 13:24:20.457487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457469    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy"
Nov 03 13:24:20.457487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457473    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy"
Nov 03 13:24:20.457644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457493    5420 cpu_manager.go:414] "containerMap: removing container" podUID="a040a8ac-bd8c-4078-b04d-c398e1f6ba9b" containerName="revoke-keys"
Nov 03 13:24:20.457644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457498    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a040a8ac-bd8c-4078-b04d-c398e1f6ba9b" containerName="revoke-keys"
Nov 03 13:24:20.457644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457604    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a040a8ac-bd8c-4078-b04d-c398e1f6ba9b" containerName="revoke-keys"
Nov 03 13:24:20.457644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457617    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy"
Nov 03 13:24:20.457644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457624    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1cd1f4-1249-4347-b5bf-f75d94de573e" containerName="istio-proxy"
Nov 03 13:24:20.457644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457630    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy"
Nov 03 13:24:20.457644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.457637    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29ba0ec-171e-44ef-96e0-e8fbd2a73e27" containerName="istio-proxy"
Nov 03 13:24:20.463181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.463154    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.468508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.468474    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:24:20.468669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.468541    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:24:20.468669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.468561    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:24:20.470177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.470119    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:24:20.471153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.471132    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"]
Nov 03 13:24:20.522715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.522654    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:20.566175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.566141    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqpk\" (UniqueName: \"kubernetes.io/projected/1fa733fd-9c23-4cd7-9998-6413aec87cdb-kube-api-access-tbqpk\") pod \"maas-api-revoke-sub-a31fe6b2-4l99j\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.566175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.566182    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a31fe6b2-4l99j\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.566438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.566206    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a31fe6b2-4l99j\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.668409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.668365    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqpk\" (UniqueName: \"kubernetes.io/projected/1fa733fd-9c23-4cd7-9998-6413aec87cdb-kube-api-access-tbqpk\") pod \"maas-api-revoke-sub-a31fe6b2-4l99j\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.668576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.668431    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a31fe6b2-4l99j\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.668576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.668471    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a31fe6b2-4l99j\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.669336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.669299    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a31fe6b2-4l99j\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.670675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.670644    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a31fe6b2-4l99j\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.679775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.679748    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqpk\" (UniqueName: \"kubernetes.io/projected/1fa733fd-9c23-4cd7-9998-6413aec87cdb-kube-api-access-tbqpk\") pod \"maas-api-revoke-sub-a31fe6b2-4l99j\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:20.782759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:20.782678    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:21.073414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:21.073365    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"]
Nov 03 13:24:21.168553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:21.168481    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:21.579146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:21.579103    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j" event={"ID":"1fa733fd-9c23-4cd7-9998-6413aec87cdb","Type":"ContainerStarted","Data":"e1f8bf557524f499345447bf98132500fbf75f8f66a04cc924760e6f5b1173d0"}
Nov 03 13:24:21.579374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:21.579168    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j" event={"ID":"1fa733fd-9c23-4cd7-9998-6413aec87cdb","Type":"ContainerStarted","Data":"90355ee519fdc6e10f8779aae09046243c013c3ff90762cde2f5e4ece015685b"}
Nov 03 13:24:21.599372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:21.599275    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j" podStartSLOduration=1.599261726 podStartE2EDuration="1.599261726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:21.597378449 +0000 UTC m=+902.675143471" watchObservedRunningTime="2026-11-03 13:24:21.599261726 +0000 UTC m=+902.677026752"
Nov 03 13:24:21.823410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:21.823381    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:22.590784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:22.590749    5420 generic.go:356] "Generic (PLEG): container finished" podID="1fa733fd-9c23-4cd7-9998-6413aec87cdb" containerID="e1f8bf557524f499345447bf98132500fbf75f8f66a04cc924760e6f5b1173d0" exitCode=0
Nov 03 13:24:22.590966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:22.590844    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j" event={"ID":"1fa733fd-9c23-4cd7-9998-6413aec87cdb","Type":"ContainerDied","Data":"e1f8bf557524f499345447bf98132500fbf75f8f66a04cc924760e6f5b1173d0"}
Nov 03 13:24:22.693568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:22.693536    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:23.307384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.307351    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:23.764921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.764898    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:23.913474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.913384    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-openshift-ca\") pod \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") "
Nov 03 13:24:23.913648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.913561    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-xks-ca\") pod \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") "
Nov 03 13:24:23.913747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.913643    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa733fd-9c23-4cd7-9998-6413aec87cdb-kube-api-access-tbqpk\" (UniqueName: \"kubernetes.io/projected/1fa733fd-9c23-4cd7-9998-6413aec87cdb-kube-api-access-tbqpk\") pod \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\" (UID: \"1fa733fd-9c23-4cd7-9998-6413aec87cdb\") "
Nov 03 13:24:23.913855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.913829    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-openshift-ca" pod "1fa733fd-9c23-4cd7-9998-6413aec87cdb" (UID: "1fa733fd-9c23-4cd7-9998-6413aec87cdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:23.913914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.913857    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:23.914192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.914164    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:23.915626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.915603    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa733fd-9c23-4cd7-9998-6413aec87cdb-kube-api-access-tbqpk" pod "1fa733fd-9c23-4cd7-9998-6413aec87cdb" (UID: "1fa733fd-9c23-4cd7-9998-6413aec87cdb"). InnerVolumeSpecName "kube-api-access-tbqpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:23.915626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:23.915611    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-xks-ca" pod "1fa733fd-9c23-4cd7-9998-6413aec87cdb" (UID: "1fa733fd-9c23-4cd7-9998-6413aec87cdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:24.015562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.015509    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa733fd-9c23-4cd7-9998-6413aec87cdb-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:24.015562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.015556    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbqpk\" (UniqueName: \"kubernetes.io/projected/1fa733fd-9c23-4cd7-9998-6413aec87cdb-kube-api-access-tbqpk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:24.524048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.524010    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"]
Nov 03 13:24:24.524781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.524758    5420 cpu_manager.go:414] "containerMap: removing container" podUID="1fa733fd-9c23-4cd7-9998-6413aec87cdb" containerName="revoke-keys"
Nov 03 13:24:24.524781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.524776    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa733fd-9c23-4cd7-9998-6413aec87cdb" containerName="revoke-keys"
Nov 03 13:24:24.524927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.524813    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy"
Nov 03 13:24:24.524927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.524821    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8968f39e-7bbd-4d21-afa6-67116c521656" containerName="istio-proxy"
Nov 03 13:24:24.525040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.525009    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa733fd-9c23-4cd7-9998-6413aec87cdb" containerName="revoke-keys"
Nov 03 13:24:24.525975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.525943    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:24.529938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.529915    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.556197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.556170    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"]
Nov 03 13:24:24.608535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.608487    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"
Nov 03 13:24:24.608720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.608482    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j" event={"ID":"1fa733fd-9c23-4cd7-9998-6413aec87cdb","Type":"ContainerDied","Data":"90355ee519fdc6e10f8779aae09046243c013c3ff90762cde2f5e4ece015685b"}
Nov 03 13:24:24.608720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.608635    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90355ee519fdc6e10f8779aae09046243c013c3ff90762cde2f5e4ece015685b"
Nov 03 13:24:24.623003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.622977    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.623129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.623039    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.623199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.623179    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85bb78e-5822-49f1-b38c-de584bd60c6b-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.623259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.623227    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.623365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.623346    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.623432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.623405    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.623509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.623468    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lznvd\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-kube-api-access-lznvd\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.623554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.623524    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.623601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.623549    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.724981    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725035    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725059    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lznvd\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-kube-api-access-lznvd\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725085    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725105    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725539    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725565    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725617    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725700    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725763    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85bb78e-5822-49f1-b38c-de584bd60c6b-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725799    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.725917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.725887    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.726178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.726161    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.726588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.726565    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85bb78e-5822-49f1-b38c-de584bd60c6b-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.727702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.727683    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.727816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.727736    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.732723 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:24:24.732690    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:24:24.733357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.733336    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.733431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.733376    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lznvd\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-kube-api-access-lznvd\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.856754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.856683    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:24.948764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:24.948727    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb" podUID="be8c0281-3962-4727-866f-3990ac17c9a8" containerName="payload-processing" containerID="cri-o://d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142" gracePeriod=25
Nov 03 13:24:25.151065 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:25.151031    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85bb78e_5822_49f1_b38c_de584bd60c6b.slice/crio-3d26daa26f01f15030bc7487426f8c9bb7ed01b1fb53693f83a43d41086a5706 WatchSource:0}: Error finding container 3d26daa26f01f15030bc7487426f8c9bb7ed01b1fb53693f83a43d41086a5706: Status 404 returned error can't find the container with id 3d26daa26f01f15030bc7487426f8c9bb7ed01b1fb53693f83a43d41086a5706
Nov 03 13:24:25.151835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.151810    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"]
Nov 03 13:24:25.154464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.154420    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:24:25.154576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.154508    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:24:25.154576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.154550    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:24:25.198806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.198783    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:24:25.316856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.316820    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:25.334914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.334886    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-kube-api-access-rx6w7\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-kube-api-access-rx6w7\") pod \"be8c0281-3962-4727-866f-3990ac17c9a8\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") "
Nov 03 13:24:25.335092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.334953    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-provider-credentials\") pod \"be8c0281-3962-4727-866f-3990ac17c9a8\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") "
Nov 03 13:24:25.335092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.335013    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-plugins-config-volume\") pod \"be8c0281-3962-4727-866f-3990ac17c9a8\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") "
Nov 03 13:24:25.335092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.335081    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-routing-overlay-volume\") pod \"be8c0281-3962-4727-866f-3990ac17c9a8\" (UID: \"be8c0281-3962-4727-866f-3990ac17c9a8\") "
Nov 03 13:24:25.335490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.335459    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-plugins-config-volume" pod "be8c0281-3962-4727-866f-3990ac17c9a8" (UID: "be8c0281-3962-4727-866f-3990ac17c9a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:25.335793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.335659    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-routing-overlay-volume" pod "be8c0281-3962-4727-866f-3990ac17c9a8" (UID: "be8c0281-3962-4727-866f-3990ac17c9a8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:25.335934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.335830    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:25.337261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.337193    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-provider-credentials" pod "be8c0281-3962-4727-866f-3990ac17c9a8" (UID: "be8c0281-3962-4727-866f-3990ac17c9a8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:25.337341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.337266    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-kube-api-access-rx6w7" pod "be8c0281-3962-4727-866f-3990ac17c9a8" (UID: "be8c0281-3962-4727-866f-3990ac17c9a8"). InnerVolumeSpecName "kube-api-access-rx6w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:25.439972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.437125    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx6w7\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-kube-api-access-rx6w7\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:25.439972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.437164    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8c0281-3962-4727-866f-3990ac17c9a8-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:25.439972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.437180    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8c0281-3962-4727-866f-3990ac17c9a8-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:25.637282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.637010    5420 generic.go:356] "Generic (PLEG): container finished" podID="be8c0281-3962-4727-866f-3990ac17c9a8" containerID="d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142" exitCode=0
Nov 03 13:24:25.637819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.637183    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb" event={"ID":"be8c0281-3962-4727-866f-3990ac17c9a8","Type":"ContainerDied","Data":"d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142"}
Nov 03 13:24:25.637819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.637262    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"
Nov 03 13:24:25.637819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.637494    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb" event={"ID":"be8c0281-3962-4727-866f-3990ac17c9a8","Type":"ContainerDied","Data":"7080f57d99d1f2f0796a9a390537e0b7b71fb79493ba66ecb311985598913ca3"}
Nov 03 13:24:25.637819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.637537    5420 scope.go:122] "RemoveContainer" containerID="d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142"
Nov 03 13:24:25.642609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.642581    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" event={"ID":"d85bb78e-5822-49f1-b38c-de584bd60c6b","Type":"ContainerStarted","Data":"f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571"}
Nov 03 13:24:25.642815 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.642794    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" event={"ID":"d85bb78e-5822-49f1-b38c-de584bd60c6b","Type":"ContainerStarted","Data":"3d26daa26f01f15030bc7487426f8c9bb7ed01b1fb53693f83a43d41086a5706"}
Nov 03 13:24:25.672425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.672404    5420 scope.go:122] "RemoveContainer" containerID="d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142"
Nov 03 13:24:25.672793 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:24:25.672774    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142\": container with ID starting with d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142 not found: ID does not exist" containerID="d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142"
Nov 03 13:24:25.672873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.672805    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142"} err="failed to get container status \"d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142\": rpc error: code = NotFound desc = could not find container \"d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142\": container with ID starting with d751e8e21e6448f163726d6fc5be50ece2cbb9fe6a43fcf7add9da53a02b9142 not found: ID does not exist"
Nov 03 13:24:25.675570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.675526    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" podStartSLOduration=1.675513356 podStartE2EDuration="1.675513356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:25.669688723 +0000 UTC m=+906.747453751" watchObservedRunningTime="2026-11-03 13:24:25.675513356 +0000 UTC m=+906.753278459"
Nov 03 13:24:25.695914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.695853    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"]
Nov 03 13:24:25.701154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.701129    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b68mtb"]
Nov 03 13:24:25.859235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:25.859038    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:26.322782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:26.322652    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:26.859338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:26.859305    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 13:24:26.859920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:26.859887    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 13:24:27.085054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:27.085020    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8c0281-3962-4727-866f-3990ac17c9a8" path="/var/lib/kubelet/pods/be8c0281-3962-4727-866f-3990ac17c9a8/volumes"
Nov 03 13:24:27.871235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:27.859550    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 13:24:27.871235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:27.859607    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 13:24:27.918235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:27.907705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:28.857899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:28.857863    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 13:24:28.858092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:28.857915    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 13:24:28.883377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:28.883347    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:29.537834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:29.537774    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:29.857108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:29.857026    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 13:24:29.857108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:29.857075    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 13:24:30.187828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.187750    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:30.379025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.378995    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"]
Nov 03 13:24:30.379675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.379659    5420 cpu_manager.go:414] "containerMap: removing container" podUID="be8c0281-3962-4727-866f-3990ac17c9a8" containerName="payload-processing"
Nov 03 13:24:30.379675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.379675    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8c0281-3962-4727-866f-3990ac17c9a8" containerName="payload-processing"
Nov 03 13:24:30.379807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.379792    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8c0281-3962-4727-866f-3990ac17c9a8" containerName="payload-processing"
Nov 03 13:24:30.385542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.385521    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.389821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.389795    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"]
Nov 03 13:24:30.389931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.389836    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:24:30.389931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.389801    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:24:30.391779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.391758    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:24:30.391900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.391855    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:24:30.481322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.481293    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d41b4d1-wzdkt\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.481476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.481462    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d41b4d1-wzdkt\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.481526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.481499    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qn4h\" (UniqueName: \"kubernetes.io/projected/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-kube-api-access-9qn4h\") pod \"maas-api-revoke-sub-4d41b4d1-wzdkt\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.582954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.582903    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d41b4d1-wzdkt\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.583131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.582991    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qn4h\" (UniqueName: \"kubernetes.io/projected/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-kube-api-access-9qn4h\") pod \"maas-api-revoke-sub-4d41b4d1-wzdkt\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.583193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.583169    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d41b4d1-wzdkt\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.583550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.583530    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d41b4d1-wzdkt\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.585304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.585290    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d41b4d1-wzdkt\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.593283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.593257    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qn4h\" (UniqueName: \"kubernetes.io/projected/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-kube-api-access-9qn4h\") pod \"maas-api-revoke-sub-4d41b4d1-wzdkt\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.703390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.703363    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:30.859711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.859303    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 13:24:30.859711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.859369    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 13:24:30.995981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:30.995938    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:31.018067 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:31.018035    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc9bd4b_8f70_4dc2_b168_8dc60745d310.slice/crio-29c8f490eb4e017423c4dfc49f2b020cf72685d172c26a3aff3fad615461a462 WatchSource:0}: Error finding container 29c8f490eb4e017423c4dfc49f2b020cf72685d172c26a3aff3fad615461a462: Status 404 returned error can't find the container with id 29c8f490eb4e017423c4dfc49f2b020cf72685d172c26a3aff3fad615461a462
Nov 03 13:24:31.019456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:31.019428    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"]
Nov 03 13:24:31.631772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:31.631740    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:31.775764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:31.775729    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt" event={"ID":"7dc9bd4b-8f70-4dc2-b168-8dc60745d310","Type":"ContainerStarted","Data":"62fe23324bdb35855813f7636b65e295eed2d686dcae330780e044c0eb1532d5"}
Nov 03 13:24:31.775923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:31.775785    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt" event={"ID":"7dc9bd4b-8f70-4dc2-b168-8dc60745d310","Type":"ContainerStarted","Data":"29c8f490eb4e017423c4dfc49f2b020cf72685d172c26a3aff3fad615461a462"}
Nov 03 13:24:31.803795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:31.803732    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt" podStartSLOduration=1.8037132809999998 podStartE2EDuration="1.803713281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:31.794432806 +0000 UTC m=+912.872197835" watchObservedRunningTime="2026-11-03 13:24:31.803713281 +0000 UTC m=+912.881478308"
Nov 03 13:24:32.255606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:32.255571    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:32.256123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:32.256100    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:32.256868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:32.256843    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:24:32.261818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:32.261796    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:32.785970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:32.785935    5420 generic.go:356] "Generic (PLEG): container finished" podID="7dc9bd4b-8f70-4dc2-b168-8dc60745d310" containerID="62fe23324bdb35855813f7636b65e295eed2d686dcae330780e044c0eb1532d5" exitCode=0
Nov 03 13:24:32.786474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:32.786020    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt" event={"ID":"7dc9bd4b-8f70-4dc2-b168-8dc60745d310","Type":"ContainerDied","Data":"62fe23324bdb35855813f7636b65e295eed2d686dcae330780e044c0eb1532d5"}
Nov 03 13:24:32.915850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:32.915820    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:33.059012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.058937    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:33.319065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.318966    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"]
Nov 03 13:24:33.325310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.325278    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.337305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.337269    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ef1693-openshift-default-dockercfg-fcmw7\""
Nov 03 13:24:33.355569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.355536    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"]
Nov 03 13:24:33.426993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.426959    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-envoy\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.427166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.427005    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b73f05-a195-4207-974f-a30c46e31ac1-istiod-ca-cert\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.427166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.427030    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-certs\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.427364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.427180    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-data\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.427364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.427229    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b73f05-a195-4207-974f-a30c46e31ac1-istio-podinfo\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.427364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.427269    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcr9\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-kube-api-access-4fcr9\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.427364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.427290    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-credential-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.427503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.427376    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.427503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.427417    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-istio-token\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.534240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.532144    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.534240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.532241    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-istio-token\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.541286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.536879    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.556231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.550904    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-envoy\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.556231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.550975    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b73f05-a195-4207-974f-a30c46e31ac1-istiod-ca-cert\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.556231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.551016    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-certs\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.556231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.551059    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-data\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.556231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.551090    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b73f05-a195-4207-974f-a30c46e31ac1-istio-podinfo\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.556231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.551156    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcr9\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-kube-api-access-4fcr9\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.556231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.551198    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-credential-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.566630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.562555    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:33.566630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.563283    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-certs\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.571135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.570780    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-credential-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.571452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.571417    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-data\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.572091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.572048    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b73f05-a195-4207-974f-a30c46e31ac1-istiod-ca-cert\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.576839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.576810    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b73f05-a195-4207-974f-a30c46e31ac1-istio-podinfo\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.582932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.582895    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-envoy\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.583444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.583422    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-istio-token\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.586704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.586640    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcr9\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-kube-api-access-4fcr9\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.656678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.656641    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:33.714672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.714605    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:33.963397 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:33.962747    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b73f05_a195_4207_974f_a30c46e31ac1.slice/crio-fb2d10c4af7f64f144439f3153104b1b8a36263ec335dcfbf5caa796cc8d69d5 WatchSource:0}: Error finding container fb2d10c4af7f64f144439f3153104b1b8a36263ec335dcfbf5caa796cc8d69d5: Status 404 returned error can't find the container with id fb2d10c4af7f64f144439f3153104b1b8a36263ec335dcfbf5caa796cc8d69d5
Nov 03 13:24:33.965866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.965826    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:24:33.965988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.965918    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:24:33.965988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.965963    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:24:33.966651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.966623    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"]
Nov 03 13:24:33.989035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:33.989010    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:34.085953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.085916    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-xks-ca\") pod \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") "
Nov 03 13:24:34.086140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.086033    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-kube-api-access-9qn4h\" (UniqueName: \"kubernetes.io/projected/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-kube-api-access-9qn4h\") pod \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") "
Nov 03 13:24:34.086140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.086129    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-openshift-ca\") pod \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\" (UID: \"7dc9bd4b-8f70-4dc2-b168-8dc60745d310\") "
Nov 03 13:24:34.086570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.086520    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-openshift-ca" pod "7dc9bd4b-8f70-4dc2-b168-8dc60745d310" (UID: "7dc9bd4b-8f70-4dc2-b168-8dc60745d310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:34.086889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.086866    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:34.088206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.088174    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-xks-ca" pod "7dc9bd4b-8f70-4dc2-b168-8dc60745d310" (UID: "7dc9bd4b-8f70-4dc2-b168-8dc60745d310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:34.088344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.088324    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-kube-api-access-9qn4h" pod "7dc9bd4b-8f70-4dc2-b168-8dc60745d310" (UID: "7dc9bd4b-8f70-4dc2-b168-8dc60745d310"). InnerVolumeSpecName "kube-api-access-9qn4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:34.188877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.188795    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qn4h\" (UniqueName: \"kubernetes.io/projected/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-kube-api-access-9qn4h\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:34.188877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.188831    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc9bd4b-8f70-4dc2-b168-8dc60745d310-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:34.271401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.271367    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:34.594013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.593985    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:34.666498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.666465    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"]
Nov 03 13:24:34.667228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.667188    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7dc9bd4b-8f70-4dc2-b168-8dc60745d310" containerName="revoke-keys"
Nov 03 13:24:34.667228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.667222    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc9bd4b-8f70-4dc2-b168-8dc60745d310" containerName="revoke-keys"
Nov 03 13:24:34.667361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.667349    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc9bd4b-8f70-4dc2-b168-8dc60745d310" containerName="revoke-keys"
Nov 03 13:24:34.671433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.671409    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.677848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.677818    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"]
Nov 03 13:24:34.799205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.799170    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc374748-mfqnk\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.799395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.799232    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc374748-mfqnk\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.799437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.799404    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldzzl\" (UniqueName: \"kubernetes.io/projected/90da7188-9990-498b-ad2e-71d477adbce9-kube-api-access-ldzzl\") pod \"maas-api-revoke-sub-fc374748-mfqnk\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.811944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.811912    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" event={"ID":"11b73f05-a195-4207-974f-a30c46e31ac1","Type":"ContainerStarted","Data":"eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243"}
Nov 03 13:24:34.812126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.811971    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" event={"ID":"11b73f05-a195-4207-974f-a30c46e31ac1","Type":"ContainerStarted","Data":"fb2d10c4af7f64f144439f3153104b1b8a36263ec335dcfbf5caa796cc8d69d5"}
Nov 03 13:24:34.813601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.813582    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"
Nov 03 13:24:34.813739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.813608    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt" event={"ID":"7dc9bd4b-8f70-4dc2-b168-8dc60745d310","Type":"ContainerDied","Data":"29c8f490eb4e017423c4dfc49f2b020cf72685d172c26a3aff3fad615461a462"}
Nov 03 13:24:34.813739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.813651    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29c8f490eb4e017423c4dfc49f2b020cf72685d172c26a3aff3fad615461a462"
Nov 03 13:24:34.838607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.838524    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" podStartSLOduration=1.838509031 podStartE2EDuration="1.838509031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:34.833691866 +0000 UTC m=+915.911456893" watchObservedRunningTime="2026-11-03 13:24:34.838509031 +0000 UTC m=+915.916274054"
Nov 03 13:24:34.900892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.900860    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldzzl\" (UniqueName: \"kubernetes.io/projected/90da7188-9990-498b-ad2e-71d477adbce9-kube-api-access-ldzzl\") pod \"maas-api-revoke-sub-fc374748-mfqnk\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.901070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.901054    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc374748-mfqnk\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.901173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.901080    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc374748-mfqnk\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.901837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.901811    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc374748-mfqnk\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.903631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.903566    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc374748-mfqnk\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.909647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.909624    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldzzl\" (UniqueName: \"kubernetes.io/projected/90da7188-9990-498b-ad2e-71d477adbce9-kube-api-access-ldzzl\") pod \"maas-api-revoke-sub-fc374748-mfqnk\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:34.988472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:34.988160    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:35.204164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.204134    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"]
Nov 03 13:24:35.216363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.216334    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"]
Nov 03 13:24:35.282294 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:35.282203    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90da7188_9990_498b_ad2e_71d477adbce9.slice/crio-b459e02e4a928d2af81807f710587b5850556f29d812b2118517c3144dd0c610 WatchSource:0}: Error finding container b459e02e4a928d2af81807f710587b5850556f29d812b2118517c3144dd0c610: Status 404 returned error can't find the container with id b459e02e4a928d2af81807f710587b5850556f29d812b2118517c3144dd0c610
Nov 03 13:24:35.283542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.283508    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"]
Nov 03 13:24:35.325468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.325440    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:35.460856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.460755    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:35.602564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.602527    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:35.657264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.657227    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:35.663071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.663044    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:35.742002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.741891    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:35.826686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.826648    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk" event={"ID":"90da7188-9990-498b-ad2e-71d477adbce9","Type":"ContainerStarted","Data":"731c3ab0468d487788b85a457456f15005a00f616bb1eea6885b1c63efdcb8a7"}
Nov 03 13:24:35.826857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.826715    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk" event={"ID":"90da7188-9990-498b-ad2e-71d477adbce9","Type":"ContainerStarted","Data":"b459e02e4a928d2af81807f710587b5850556f29d812b2118517c3144dd0c610"}
Nov 03 13:24:35.827191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.827164    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:35.828314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.828292    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:24:35.847886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:35.847833    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk" podStartSLOduration=1.847818491 podStartE2EDuration="1.847818491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:35.845179931 +0000 UTC m=+916.922944961" watchObservedRunningTime="2026-11-03 13:24:35.847818491 +0000 UTC m=+916.925583517"
Nov 03 13:24:36.272136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:36.272089    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:36.419522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:36.419439    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:36.837975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:36.837943    5420 generic.go:356] "Generic (PLEG): container finished" podID="90da7188-9990-498b-ad2e-71d477adbce9" containerID="731c3ab0468d487788b85a457456f15005a00f616bb1eea6885b1c63efdcb8a7" exitCode=0
Nov 03 13:24:36.838136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:36.838070    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk" event={"ID":"90da7188-9990-498b-ad2e-71d477adbce9","Type":"ContainerDied","Data":"731c3ab0468d487788b85a457456f15005a00f616bb1eea6885b1c63efdcb8a7"}
Nov 03 13:24:37.209933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:37.209892    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:37.361251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:37.361191    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:37.989083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:37.989050    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:38.025759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.023321    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:38.150158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.150068    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90da7188-9990-498b-ad2e-71d477adbce9-kube-api-access-ldzzl\" (UniqueName: \"kubernetes.io/projected/90da7188-9990-498b-ad2e-71d477adbce9-kube-api-access-ldzzl\") pod \"90da7188-9990-498b-ad2e-71d477adbce9\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") "
Nov 03 13:24:38.150158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.150106    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-xks-ca\") pod \"90da7188-9990-498b-ad2e-71d477adbce9\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") "
Nov 03 13:24:38.150158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.150141    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-openshift-ca\") pod \"90da7188-9990-498b-ad2e-71d477adbce9\" (UID: \"90da7188-9990-498b-ad2e-71d477adbce9\") "
Nov 03 13:24:38.150632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.150597    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-openshift-ca" pod "90da7188-9990-498b-ad2e-71d477adbce9" (UID: "90da7188-9990-498b-ad2e-71d477adbce9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:38.150924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.150903    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:38.152390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.152363    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90da7188-9990-498b-ad2e-71d477adbce9-kube-api-access-ldzzl" pod "90da7188-9990-498b-ad2e-71d477adbce9" (UID: "90da7188-9990-498b-ad2e-71d477adbce9"). InnerVolumeSpecName "kube-api-access-ldzzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:38.152486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.152437    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-xks-ca" pod "90da7188-9990-498b-ad2e-71d477adbce9" (UID: "90da7188-9990-498b-ad2e-71d477adbce9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:38.252186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.252150    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldzzl\" (UniqueName: \"kubernetes.io/projected/90da7188-9990-498b-ad2e-71d477adbce9-kube-api-access-ldzzl\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:38.252186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.252180    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da7188-9990-498b-ad2e-71d477adbce9-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:38.541281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.541248    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:38.584782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.584741    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"]
Nov 03 13:24:38.587375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.587344    5420 cpu_manager.go:414] "containerMap: removing container" podUID="90da7188-9990-498b-ad2e-71d477adbce9" containerName="revoke-keys"
Nov 03 13:24:38.587375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.587369    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90da7188-9990-498b-ad2e-71d477adbce9" containerName="revoke-keys"
Nov 03 13:24:38.587795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.587777    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="90da7188-9990-498b-ad2e-71d477adbce9" containerName="revoke-keys"
Nov 03 13:24:38.598473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.598445    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:38.602044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.602016    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"]
Nov 03 13:24:38.603281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.603248    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ef1693-dockercfg-m77vm\""
Nov 03 13:24:38.603474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.603456    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ef1693\""
Nov 03 13:24:38.656976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.656946    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a4b363-0733-47bc-a0cb-69e56524d691-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd\" (UID: \"c7a4b363-0733-47bc-a0cb-69e56524d691\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:38.657112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.656992    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwmh\" (UniqueName: \"kubernetes.io/projected/c7a4b363-0733-47bc-a0cb-69e56524d691-kube-api-access-bdwmh\") pod \"payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd\" (UID: \"c7a4b363-0733-47bc-a0cb-69e56524d691\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:38.685678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.685652    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:38.737893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.720424    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"]
Nov 03 13:24:38.762718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.761418    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a4b363-0733-47bc-a0cb-69e56524d691-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd\" (UID: \"c7a4b363-0733-47bc-a0cb-69e56524d691\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:38.762718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.761485    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwmh\" (UniqueName: \"kubernetes.io/projected/c7a4b363-0733-47bc-a0cb-69e56524d691-kube-api-access-bdwmh\") pod \"payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd\" (UID: \"c7a4b363-0733-47bc-a0cb-69e56524d691\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:38.766792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.766759    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a4b363-0733-47bc-a0cb-69e56524d691-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd\" (UID: \"c7a4b363-0733-47bc-a0cb-69e56524d691\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:38.768721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.768088    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"]
Nov 03 13:24:38.768721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.768265    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:38.781506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.781204    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwmh\" (UniqueName: \"kubernetes.io/projected/c7a4b363-0733-47bc-a0cb-69e56524d691-kube-api-access-bdwmh\") pod \"payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd\" (UID: \"c7a4b363-0733-47bc-a0cb-69e56524d691\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:38.863340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.863307    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c153d-e665-4083-8848-94bf3fa8fe31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk\" (UID: \"479c153d-e665-4083-8848-94bf3fa8fe31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:38.863514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.863437    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg2t5\" (UniqueName: \"kubernetes.io/projected/479c153d-e665-4083-8848-94bf3fa8fe31-kube-api-access-lg2t5\") pod \"payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk\" (UID: \"479c153d-e665-4083-8848-94bf3fa8fe31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:38.868273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.868243    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk" event={"ID":"90da7188-9990-498b-ad2e-71d477adbce9","Type":"ContainerDied","Data":"b459e02e4a928d2af81807f710587b5850556f29d812b2118517c3144dd0c610"}
Nov 03 13:24:38.868505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.868300    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b459e02e4a928d2af81807f710587b5850556f29d812b2118517c3144dd0c610"
Nov 03 13:24:38.868505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.868372    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"
Nov 03 13:24:38.913360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.913326    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:38.923235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.917162    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:38.965419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.965383    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg2t5\" (UniqueName: \"kubernetes.io/projected/479c153d-e665-4083-8848-94bf3fa8fe31-kube-api-access-lg2t5\") pod \"payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk\" (UID: \"479c153d-e665-4083-8848-94bf3fa8fe31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:38.965609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.965588    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c153d-e665-4083-8848-94bf3fa8fe31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk\" (UID: \"479c153d-e665-4083-8848-94bf3fa8fe31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:38.966772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.966605    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c153d-e665-4083-8848-94bf3fa8fe31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk\" (UID: \"479c153d-e665-4083-8848-94bf3fa8fe31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:38.975478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:38.975438    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg2t5\" (UniqueName: \"kubernetes.io/projected/479c153d-e665-4083-8848-94bf3fa8fe31-kube-api-access-lg2t5\") pod \"payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk\" (UID: \"479c153d-e665-4083-8848-94bf3fa8fe31\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:39.119095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.119004    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:39.217794 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:39.217759    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a4b363_0733_47bc_a0cb_69e56524d691.slice/crio-76b5814d0e3973b2435c9c6b4973243c914598bd52933b5ca8ef45dc1314b53e WatchSource:0}: Error finding container 76b5814d0e3973b2435c9c6b4973243c914598bd52933b5ca8ef45dc1314b53e: Status 404 returned error can't find the container with id 76b5814d0e3973b2435c9c6b4973243c914598bd52933b5ca8ef45dc1314b53e
Nov 03 13:24:39.219600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.219572    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"]
Nov 03 13:24:39.403737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:39.403708    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479c153d_e665_4083_8848_94bf3fa8fe31.slice/crio-2607ee26ce5109dde2456edb9ddecbf6e026294a279c38682393831991f8517b WatchSource:0}: Error finding container 2607ee26ce5109dde2456edb9ddecbf6e026294a279c38682393831991f8517b: Status 404 returned error can't find the container with id 2607ee26ce5109dde2456edb9ddecbf6e026294a279c38682393831991f8517b
Nov 03 13:24:39.406567 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.406547    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"]
Nov 03 13:24:39.713801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.713759    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:39.878067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.878031    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd" event={"ID":"c7a4b363-0733-47bc-a0cb-69e56524d691","Type":"ContainerStarted","Data":"258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95"}
Nov 03 13:24:39.878248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.878081    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd" event={"ID":"c7a4b363-0733-47bc-a0cb-69e56524d691","Type":"ContainerStarted","Data":"76b5814d0e3973b2435c9c6b4973243c914598bd52933b5ca8ef45dc1314b53e"}
Nov 03 13:24:39.878248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.878129    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:39.879782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.879760    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk" event={"ID":"479c153d-e665-4083-8848-94bf3fa8fe31","Type":"ContainerStarted","Data":"b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf"}
Nov 03 13:24:39.879887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.879799    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk" event={"ID":"479c153d-e665-4083-8848-94bf3fa8fe31","Type":"ContainerStarted","Data":"2607ee26ce5109dde2456edb9ddecbf6e026294a279c38682393831991f8517b"}
Nov 03 13:24:39.879887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.879872    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:39.898848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.898786    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd" podStartSLOduration=1.898774264 podStartE2EDuration="1.898774264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:39.896146377 +0000 UTC m=+920.973911416" watchObservedRunningTime="2026-11-03 13:24:39.898774264 +0000 UTC m=+920.976539290"
Nov 03 13:24:39.918412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:39.918372    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk" podStartSLOduration=1.9183596349999998 podStartE2EDuration="1.918359635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:39.917569673 +0000 UTC m=+920.995334699" watchObservedRunningTime="2026-11-03 13:24:39.918359635 +0000 UTC m=+920.996124681"
Nov 03 13:24:40.204594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.204528    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9" podUID="284d1e11-52e5-4c88-925a-35d22f1239d9" containerName="payload-processing" containerID="cri-o://f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14" gracePeriod=25
Nov 03 13:24:40.217238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.217185    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz" podUID="263fca38-ec96-41fe-9a14-5346ffd41192" containerName="payload-pre-processing" containerID="cri-o://336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80" gracePeriod=25
Nov 03 13:24:40.288689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.288664    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:40.590453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.590375    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"]
Nov 03 13:24:40.595097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.595066    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.599393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.599357    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:24:40.600923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.600896    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:24:40.600923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.600896    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:24:40.601107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.600990    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:24:40.601806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.601782    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"]
Nov 03 13:24:40.690681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.690646    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdl2k\" (UniqueName: \"kubernetes.io/projected/72b9c406-e782-41c5-99c8-2887cf33bed3-kube-api-access-mdl2k\") pod \"maas-api-revoke-sub-d0055b7d-b9x8d\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.690817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.690688    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0055b7d-b9x8d\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.690956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.690936    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0055b7d-b9x8d\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.793618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.793576    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0055b7d-b9x8d\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.794051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.793708    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdl2k\" (UniqueName: \"kubernetes.io/projected/72b9c406-e782-41c5-99c8-2887cf33bed3-kube-api-access-mdl2k\") pod \"maas-api-revoke-sub-d0055b7d-b9x8d\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.794051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.793745    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0055b7d-b9x8d\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.796840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.796808    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0055b7d-b9x8d\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.797506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.797479    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0055b7d-b9x8d\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.806980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.806947    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdl2k\" (UniqueName: \"kubernetes.io/projected/72b9c406-e782-41c5-99c8-2887cf33bed3-kube-api-access-mdl2k\") pod \"maas-api-revoke-sub-d0055b7d-b9x8d\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:40.912231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:40.912136    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:41.029278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.029245    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"]
Nov 03 13:24:41.034660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.034633    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e8d6-49bmt"]
Nov 03 13:24:41.079820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.079790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:41.085629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.085594    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977ab5f9-3315-417e-9c00-02e4c38a012e" path="/var/lib/kubelet/pods/977ab5f9-3315-417e-9c00-02e4c38a012e/volumes"
Nov 03 13:24:41.173722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.173638    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:24:41.201109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.201083    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"]
Nov 03 13:24:41.895961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.895919    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:41.897807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.897779    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d" event={"ID":"72b9c406-e782-41c5-99c8-2887cf33bed3","Type":"ContainerStarted","Data":"bcaa0d2658c8a9340b477a21990955ae2cc6bc5f7589f4da502cc0288e1f0d00"}
Nov 03 13:24:41.897959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.897863    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d" event={"ID":"72b9c406-e782-41c5-99c8-2887cf33bed3","Type":"ContainerStarted","Data":"f374770fe9f64634604a3f3c8ef37410df80ec74786baf3c902d7f5048061c59"}
Nov 03 13:24:41.918962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:41.918915    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d" podStartSLOduration=1.918901007 podStartE2EDuration="1.918901007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:41.914252694 +0000 UTC m=+922.992017720" watchObservedRunningTime="2026-11-03 13:24:41.918901007 +0000 UTC m=+922.996666033"
Nov 03 13:24:42.119233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:42.116831    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:42.945235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:42.944540    5420 generic.go:356] "Generic (PLEG): container finished" podID="72b9c406-e782-41c5-99c8-2887cf33bed3" containerID="bcaa0d2658c8a9340b477a21990955ae2cc6bc5f7589f4da502cc0288e1f0d00" exitCode=0
Nov 03 13:24:42.945235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:42.944620    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d" event={"ID":"72b9c406-e782-41c5-99c8-2887cf33bed3","Type":"ContainerDied","Data":"bcaa0d2658c8a9340b477a21990955ae2cc6bc5f7589f4da502cc0288e1f0d00"}
Nov 03 13:24:43.062235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:43.057667    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:43.216241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:43.211474    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:43.975644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:43.975609    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:44.163298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.163269    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:44.210192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.210114    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-xks-ca\") pod \"72b9c406-e782-41c5-99c8-2887cf33bed3\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") "
Nov 03 13:24:44.210192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.210189    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-openshift-ca\") pod \"72b9c406-e782-41c5-99c8-2887cf33bed3\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") "
Nov 03 13:24:44.210442 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.210295    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b9c406-e782-41c5-99c8-2887cf33bed3-kube-api-access-mdl2k\" (UniqueName: \"kubernetes.io/projected/72b9c406-e782-41c5-99c8-2887cf33bed3-kube-api-access-mdl2k\") pod \"72b9c406-e782-41c5-99c8-2887cf33bed3\" (UID: \"72b9c406-e782-41c5-99c8-2887cf33bed3\") "
Nov 03 13:24:44.210580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.210554    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-openshift-ca" pod "72b9c406-e782-41c5-99c8-2887cf33bed3" (UID: "72b9c406-e782-41c5-99c8-2887cf33bed3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:44.210726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.210707    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:44.212404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.212336    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b9c406-e782-41c5-99c8-2887cf33bed3-kube-api-access-mdl2k" pod "72b9c406-e782-41c5-99c8-2887cf33bed3" (UID: "72b9c406-e782-41c5-99c8-2887cf33bed3"). InnerVolumeSpecName "kube-api-access-mdl2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:44.212541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.212446    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-xks-ca" pod "72b9c406-e782-41c5-99c8-2887cf33bed3" (UID: "72b9c406-e782-41c5-99c8-2887cf33bed3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:44.312015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.311983    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdl2k\" (UniqueName: \"kubernetes.io/projected/72b9c406-e782-41c5-99c8-2887cf33bed3-kube-api-access-mdl2k\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:44.312015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.312011    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b9c406-e782-41c5-99c8-2887cf33bed3-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:44.322196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.322164    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:44.911042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.911011    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:44.967858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.967816    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d" event={"ID":"72b9c406-e782-41c5-99c8-2887cf33bed3","Type":"ContainerDied","Data":"f374770fe9f64634604a3f3c8ef37410df80ec74786baf3c902d7f5048061c59"}
Nov 03 13:24:44.968033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.967910    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f374770fe9f64634604a3f3c8ef37410df80ec74786baf3c902d7f5048061c59"
Nov 03 13:24:44.968033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:44.967987    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"
Nov 03 13:24:45.069382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:45.069350    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9" podUID="284d1e11-52e5-4c88-925a-35d22f1239d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:24:45.069790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:45.069350    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz" podUID="263fca38-ec96-41fe-9a14-5346ffd41192" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:24:45.699492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:45.699456    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:46.384740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:46.384705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:46.584020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:46.583987    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:47.095033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.095004    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:47.636940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.636902    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"]
Nov 03 13:24:47.637835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.637814    5420 cpu_manager.go:414] "containerMap: removing container" podUID="72b9c406-e782-41c5-99c8-2887cf33bed3" containerName="revoke-keys"
Nov 03 13:24:47.637885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.637838    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b9c406-e782-41c5-99c8-2887cf33bed3" containerName="revoke-keys"
Nov 03 13:24:47.637990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.637978    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b9c406-e782-41c5-99c8-2887cf33bed3" containerName="revoke-keys"
Nov 03 13:24:47.641915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.641890    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.648135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.648074    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"kube-root-ca.crt\""
Nov 03 13:24:47.648135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.648111    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c192b\""
Nov 03 13:24:47.648135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.648113    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"openshift-service-ca.crt\""
Nov 03 13:24:47.648369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.648167    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"routing-overlay\""
Nov 03 13:24:47.649669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.649646    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-29dd648f7d07\""
Nov 03 13:24:47.649770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.649711    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-384e44d183e3\""
Nov 03 13:24:47.649770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.649722    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-06e7699b6ab0\""
Nov 03 13:24:47.649770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.649725    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-35f1d38599a0\""
Nov 03 13:24:47.649883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.649795    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-f5ffe280c5d9\""
Nov 03 13:24:47.650028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.650011    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-e2e--cd1da41a-dockercfg-txc88\""
Nov 03 13:24:47.651935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.651917    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"]
Nov 03 13:24:47.767673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.767631    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.767887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.767869    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.767947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.767929    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.767992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.767984    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfkbm\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-kube-api-access-dfkbm\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.790833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.790805    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:47.869203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.869162    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.869466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.869443    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.869568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.869522    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.869568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.869558    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfkbm\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-kube-api-access-dfkbm\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.870557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.870307    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.870557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.870334    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.873112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.873086    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.878527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.878502    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfkbm\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-kube-api-access-dfkbm\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:47.965654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:47.965620    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:48.251929 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:24:48.251895    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e3e558_7d4a_420e_bdad_8f95e5c220b8.slice/crio-a57c0c0d6ec3dad3c5d6cf941076dbb3e8fb6109e25c331d591b6bb7ddfaaec3 WatchSource:0}: Error finding container a57c0c0d6ec3dad3c5d6cf941076dbb3e8fb6109e25c331d591b6bb7ddfaaec3: Status 404 returned error can't find the container with id a57c0c0d6ec3dad3c5d6cf941076dbb3e8fb6109e25c331d591b6bb7ddfaaec3
Nov 03 13:24:48.253322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:48.253296    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"]
Nov 03 13:24:48.326951 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:48.326873    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:48.462314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:48.462243    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:49.016446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:49.016400    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw" event={"ID":"91e3e558-7d4a-420e-bdad-8f95e5c220b8","Type":"ContainerStarted","Data":"3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f"}
Nov 03 13:24:49.016925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:49.016494    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:24:49.016925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:49.016526    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw" event={"ID":"91e3e558-7d4a-420e-bdad-8f95e5c220b8","Type":"ContainerStarted","Data":"a57c0c0d6ec3dad3c5d6cf941076dbb3e8fb6109e25c331d591b6bb7ddfaaec3"}
Nov 03 13:24:49.063351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:49.063302    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw" podStartSLOduration=2.063288128 podStartE2EDuration="2.063288128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:49.051583498 +0000 UTC m=+930.129348525" watchObservedRunningTime="2026-11-03 13:24:49.063288128 +0000 UTC m=+930.141053154"
Nov 03 13:24:49.102314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:49.102286    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:49.656902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:49.656873    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:50.214756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:50.213477    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:50.890738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:50.890713    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:24:50.890940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:50.890775    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:24:50.913096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:50.913064    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:51.024861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:51.024827    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"]
Nov 03 13:24:51.031487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:51.031464    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a21fb-vnx2r"]
Nov 03 13:24:51.089042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:51.089009    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e481f0-0b30-4881-87ac-e0e96a798411" path="/var/lib/kubelet/pods/23e481f0-0b30-4881-87ac-e0e96a798411/volumes"
Nov 03 13:24:51.679052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:51.679013    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:52.423269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:52.423179    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:53.028171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:53.028141    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:53.045377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:53.045350    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"]
Nov 03 13:24:53.051338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:53.051304    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169ee085-gfxqg"]
Nov 03 13:24:53.085946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:53.085911    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06cacf08-f842-4b04-b873-e55a01365aa5" path="/var/lib/kubelet/pods/06cacf08-f842-4b04-b873-e55a01365aa5/volumes"
Nov 03 13:24:53.610529 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:53.610500    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:53.858948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:53.858900    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:54.361846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:54.361817    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:54.618707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:54.618622    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:54.731958 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:24:54.731929    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:24:54.835882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:54.835846    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:55.015291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:55.013283    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:55.068825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:55.068787    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9" podUID="284d1e11-52e5-4c88-925a-35d22f1239d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:24:55.069013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:55.068822    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz" podUID="263fca38-ec96-41fe-9a14-5346ffd41192" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:24:55.258857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:55.258827    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"]
Nov 03 13:24:55.259138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:55.259105    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" containerName="istio-proxy" containerID="cri-o://a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097" gracePeriod=30
Nov 03 13:24:55.389045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:55.388887    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:55.562813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:55.562774    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:55.709755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:55.709722    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:56.236775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:56.236734    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:56.959119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:56.959055    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:57.433310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:57.433201    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:57.578119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:57.578089    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:58.085391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:58.085355    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:58.702812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:58.702775    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:58.969175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:58.969086    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:59.506433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:59.506393    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:59.895066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:24:59.894962    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:00.087400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.087359    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:00.097263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.094410    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:25:00.699700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.699670    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:25:00.713187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.713162    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:25:00.716654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.716635    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:25:00.860427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860328    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31b93da-47a4-45cf-a17d-836371568cfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b93da-47a4-45cf-a17d-836371568cfc-istiod-ca-cert\") pod \"d31b93da-47a4-45cf-a17d-836371568cfc\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") "
Nov 03 13:25:00.860612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860507    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-kube-api-access-j9ln9\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-kube-api-access-j9ln9\") pod \"d31b93da-47a4-45cf-a17d-836371568cfc\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") "
Nov 03 13:25:00.860612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860556    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d31b93da-47a4-45cf-a17d-836371568cfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b93da-47a4-45cf-a17d-836371568cfc-istio-podinfo\") pod \"d31b93da-47a4-45cf-a17d-836371568cfc\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") "
Nov 03 13:25:00.860612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860601    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263fca38-ec96-41fe-9a14-5346ffd41192-kube-api-access-5qqjj\" (UniqueName: \"kubernetes.io/projected/263fca38-ec96-41fe-9a14-5346ffd41192-kube-api-access-5qqjj\") pod \"263fca38-ec96-41fe-9a14-5346ffd41192\" (UID: \"263fca38-ec96-41fe-9a14-5346ffd41192\") "
Nov 03 13:25:00.860783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860680    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263fca38-ec96-41fe-9a14-5346ffd41192-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263fca38-ec96-41fe-9a14-5346ffd41192-plugins-config-volume\") pod \"263fca38-ec96-41fe-9a14-5346ffd41192\" (UID: \"263fca38-ec96-41fe-9a14-5346ffd41192\") "
Nov 03 13:25:00.860783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860764    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-data\") pod \"d31b93da-47a4-45cf-a17d-836371568cfc\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") "
Nov 03 13:25:00.860883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860801    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-socket\") pod \"d31b93da-47a4-45cf-a17d-836371568cfc\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") "
Nov 03 13:25:00.860883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860837    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-istio-token\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-istio-token\") pod \"d31b93da-47a4-45cf-a17d-836371568cfc\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") "
Nov 03 13:25:00.861081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860920    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-envoy\") pod \"d31b93da-47a4-45cf-a17d-836371568cfc\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") "
Nov 03 13:25:00.861081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860955    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284d1e11-52e5-4c88-925a-35d22f1239d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284d1e11-52e5-4c88-925a-35d22f1239d9-plugins-config-volume\") pod \"284d1e11-52e5-4c88-925a-35d22f1239d9\" (UID: \"284d1e11-52e5-4c88-925a-35d22f1239d9\") "
Nov 03 13:25:00.861081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.860989    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-certs\") pod \"d31b93da-47a4-45cf-a17d-836371568cfc\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") "
Nov 03 13:25:00.861081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.861056    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284d1e11-52e5-4c88-925a-35d22f1239d9-kube-api-access-ttn69\" (UniqueName: \"kubernetes.io/projected/284d1e11-52e5-4c88-925a-35d22f1239d9-kube-api-access-ttn69\") pod \"284d1e11-52e5-4c88-925a-35d22f1239d9\" (UID: \"284d1e11-52e5-4c88-925a-35d22f1239d9\") "
Nov 03 13:25:00.861314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.861132    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-credential-socket\") pod \"d31b93da-47a4-45cf-a17d-836371568cfc\" (UID: \"d31b93da-47a4-45cf-a17d-836371568cfc\") "
Nov 03 13:25:00.862153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.862124    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-credential-socket" pod "d31b93da-47a4-45cf-a17d-836371568cfc" (UID: "d31b93da-47a4-45cf-a17d-836371568cfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:00.862671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.862648    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284d1e11-52e5-4c88-925a-35d22f1239d9-plugins-config-volume" pod "284d1e11-52e5-4c88-925a-35d22f1239d9" (UID: "284d1e11-52e5-4c88-925a-35d22f1239d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:00.863102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.863073    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-certs" pod "d31b93da-47a4-45cf-a17d-836371568cfc" (UID: "d31b93da-47a4-45cf-a17d-836371568cfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:00.863338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.863314    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31b93da-47a4-45cf-a17d-836371568cfc-istiod-ca-cert" pod "d31b93da-47a4-45cf-a17d-836371568cfc" (UID: "d31b93da-47a4-45cf-a17d-836371568cfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:00.863638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.863617    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-data" pod "d31b93da-47a4-45cf-a17d-836371568cfc" (UID: "d31b93da-47a4-45cf-a17d-836371568cfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:00.865952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.865930    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-socket" pod "d31b93da-47a4-45cf-a17d-836371568cfc" (UID: "d31b93da-47a4-45cf-a17d-836371568cfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:00.866164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.866148    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263fca38-ec96-41fe-9a14-5346ffd41192-plugins-config-volume" pod "263fca38-ec96-41fe-9a14-5346ffd41192" (UID: "263fca38-ec96-41fe-9a14-5346ffd41192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:00.867448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.867302    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-envoy" pod "d31b93da-47a4-45cf-a17d-836371568cfc" (UID: "d31b93da-47a4-45cf-a17d-836371568cfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:00.867715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.867691    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d31b93da-47a4-45cf-a17d-836371568cfc-istio-podinfo" pod "d31b93da-47a4-45cf-a17d-836371568cfc" (UID: "d31b93da-47a4-45cf-a17d-836371568cfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:25:00.867814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.867797    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-kube-api-access-j9ln9" pod "d31b93da-47a4-45cf-a17d-836371568cfc" (UID: "d31b93da-47a4-45cf-a17d-836371568cfc"). InnerVolumeSpecName "kube-api-access-j9ln9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:00.867879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.867864    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284d1e11-52e5-4c88-925a-35d22f1239d9-kube-api-access-ttn69" pod "284d1e11-52e5-4c88-925a-35d22f1239d9" (UID: "284d1e11-52e5-4c88-925a-35d22f1239d9"). InnerVolumeSpecName "kube-api-access-ttn69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:00.868134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.868110    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:00.869198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.869177    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-istio-token" pod "d31b93da-47a4-45cf-a17d-836371568cfc" (UID: "d31b93da-47a4-45cf-a17d-836371568cfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:00.869390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.869361    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263fca38-ec96-41fe-9a14-5346ffd41192-kube-api-access-5qqjj" pod "263fca38-ec96-41fe-9a14-5346ffd41192" (UID: "263fca38-ec96-41fe-9a14-5346ffd41192"). InnerVolumeSpecName "kube-api-access-5qqjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:00.963391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963360    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963388    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284d1e11-52e5-4c88-925a-35d22f1239d9-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963399    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963411    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttn69\" (UniqueName: \"kubernetes.io/projected/284d1e11-52e5-4c88-925a-35d22f1239d9-kube-api-access-ttn69\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963427    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963437    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31b93da-47a4-45cf-a17d-836371568cfc-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963449    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9ln9\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-kube-api-access-j9ln9\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963463    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31b93da-47a4-45cf-a17d-836371568cfc-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963475    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qqjj\" (UniqueName: \"kubernetes.io/projected/263fca38-ec96-41fe-9a14-5346ffd41192-kube-api-access-5qqjj\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963484    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263fca38-ec96-41fe-9a14-5346ffd41192-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963497    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963511    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31b93da-47a4-45cf-a17d-836371568cfc-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:00.963718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:00.963519    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31b93da-47a4-45cf-a17d-836371568cfc-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:01.408842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.408806    5420 generic.go:356] "Generic (PLEG): container finished" podID="d31b93da-47a4-45cf-a17d-836371568cfc" containerID="a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097" exitCode=0
Nov 03 13:25:01.409039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.408855    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"
Nov 03 13:25:01.409039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.408886    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" event={"ID":"d31b93da-47a4-45cf-a17d-836371568cfc","Type":"ContainerDied","Data":"a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097"}
Nov 03 13:25:01.409039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.408940    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn" event={"ID":"d31b93da-47a4-45cf-a17d-836371568cfc","Type":"ContainerDied","Data":"0d9c3f85ba6dee479c098763c89c70d241dfca0d2695eacdf59192dbcf80fed0"}
Nov 03 13:25:01.409039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.408962    5420 scope.go:122] "RemoveContainer" containerID="a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097"
Nov 03 13:25:01.410788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.410604    5420 generic.go:356] "Generic (PLEG): container finished" podID="263fca38-ec96-41fe-9a14-5346ffd41192" containerID="336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80" exitCode=0
Nov 03 13:25:01.410788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.410642    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"
Nov 03 13:25:01.410788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.410729    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz" event={"ID":"263fca38-ec96-41fe-9a14-5346ffd41192","Type":"ContainerDied","Data":"336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80"}
Nov 03 13:25:01.410965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.410826    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz" event={"ID":"263fca38-ec96-41fe-9a14-5346ffd41192","Type":"ContainerDied","Data":"43e431921ab0116a9328374eba3aad2a68d3dc53548ed0f7c85a25654db113e7"}
Nov 03 13:25:01.412879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.412845    5420 generic.go:356] "Generic (PLEG): container finished" podID="284d1e11-52e5-4c88-925a-35d22f1239d9" containerID="f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14" exitCode=0
Nov 03 13:25:01.413078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.413059    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9" event={"ID":"284d1e11-52e5-4c88-925a-35d22f1239d9","Type":"ContainerDied","Data":"f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14"}
Nov 03 13:25:01.413176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.413106    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9" event={"ID":"284d1e11-52e5-4c88-925a-35d22f1239d9","Type":"ContainerDied","Data":"bbbfb3982ab015b2a05b5a8a04324214ef524028cccc1a0dde337f8935512a33"}
Nov 03 13:25:01.413176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.413170    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"
Nov 03 13:25:01.451285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.445482    5420 scope.go:122] "RemoveContainer" containerID="a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097"
Nov 03 13:25:01.451285 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:25:01.446013    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097\": container with ID starting with a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097 not found: ID does not exist" containerID="a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097"
Nov 03 13:25:01.451285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.446047    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097"} err="failed to get container status \"a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097\": rpc error: code = NotFound desc = could not find container \"a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097\": container with ID starting with a27874a8f5c513492bc017a61c889f95b1d5ce1e2c824b7adeac6f26f1e1a097 not found: ID does not exist"
Nov 03 13:25:01.451285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.446070    5420 scope.go:122] "RemoveContainer" containerID="336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80"
Nov 03 13:25:01.468719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.468684    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"]
Nov 03 13:25:01.472752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.472721    5420 scope.go:122] "RemoveContainer" containerID="336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80"
Nov 03 13:25:01.473348 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:25:01.473316    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80\": container with ID starting with 336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80 not found: ID does not exist" containerID="336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80"
Nov 03 13:25:01.473473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.473377    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80"} err="failed to get container status \"336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80\": rpc error: code = NotFound desc = could not find container \"336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80\": container with ID starting with 336afd96bb8fa5b0bdcfff78cf64536faf0bde30026732365264d2464c4f7c80 not found: ID does not exist"
Nov 03 13:25:01.473473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.473411    5420 scope.go:122] "RemoveContainer" containerID="f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14"
Nov 03 13:25:01.478319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.478287    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-8fcdn"]
Nov 03 13:25:01.482542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.482516    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:01.486757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.486731    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"]
Nov 03 13:25:01.489086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.489065    5420 scope.go:122] "RemoveContainer" containerID="f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14"
Nov 03 13:25:01.489515 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:25:01.489482    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14\": container with ID starting with f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14 not found: ID does not exist" containerID="f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14"
Nov 03 13:25:01.489615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.489525    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14"} err="failed to get container status \"f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14\": rpc error: code = NotFound desc = could not find container \"f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14\": container with ID starting with f2f0d2d21ed7b58cf1f729eaff860c18f96f9da49629b53747cd4627a46f4e14 not found: ID does not exist"
Nov 03 13:25:01.495986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.495922    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a2ccd8-7f74875bbc-jrcdz"]
Nov 03 13:25:01.505286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.505253    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"]
Nov 03 13:25:01.514831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.514026    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2ccd8-69fbbfb657-2jvf9"]
Nov 03 13:25:01.740955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.735718    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:01.906294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:01.906262    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:02.112483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.112402    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:02.259746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.259719    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:02.554881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.554841    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"]
Nov 03 13:25:02.555751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.555727    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" containerName="istio-proxy"
Nov 03 13:25:02.555751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.555751    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" containerName="istio-proxy"
Nov 03 13:25:02.555938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.555760    5420 cpu_manager.go:414] "containerMap: removing container" podUID="284d1e11-52e5-4c88-925a-35d22f1239d9" containerName="payload-processing"
Nov 03 13:25:02.555938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.555765    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284d1e11-52e5-4c88-925a-35d22f1239d9" containerName="payload-processing"
Nov 03 13:25:02.555938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.555789    5420 cpu_manager.go:414] "containerMap: removing container" podUID="263fca38-ec96-41fe-9a14-5346ffd41192" containerName="payload-pre-processing"
Nov 03 13:25:02.555938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.555795    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263fca38-ec96-41fe-9a14-5346ffd41192" containerName="payload-pre-processing"
Nov 03 13:25:02.555938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.555910    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="263fca38-ec96-41fe-9a14-5346ffd41192" containerName="payload-pre-processing"
Nov 03 13:25:02.555938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.555926    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" containerName="istio-proxy"
Nov 03 13:25:02.555938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.555932    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="284d1e11-52e5-4c88-925a-35d22f1239d9" containerName="payload-processing"
Nov 03 13:25:02.560968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.560943    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.564708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.564681    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"]
Nov 03 13:25:02.610738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.610690    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-data\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.610738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.610741    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-podinfo\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.611003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.610872    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-certs\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.611003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.610974    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-envoy\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.611108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.611007    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6bm\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-kube-api-access-cg6bm\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.611108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.611036    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-credential-socket\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.611176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.611143    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-socket\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.611250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.611186    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-token\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.611331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.611313    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istiod-ca-cert\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.618316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.618292    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:02.651355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.650868    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-021a5a5c-openshift-default-dockercfg-lxqml\""
Nov 03 13:25:02.713249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.712757    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-certs\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.713249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.712840    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-envoy\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.713249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.712875    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6bm\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-kube-api-access-cg6bm\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.713249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.712913    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-credential-socket\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.713249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.712971    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-socket\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.713249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.713035    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-token\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.713249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.713098    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istiod-ca-cert\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.713249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.713223    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-data\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.713249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.713259    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-podinfo\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.714493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.714139    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-credential-socket\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.714493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.714427    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-certs\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.715370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.715341    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-socket\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.715686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.715663    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-data\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.716837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.716814    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istiod-ca-cert\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.786230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.782337    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6bm\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-kube-api-access-cg6bm\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.786230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.783293    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:02.786230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.784781    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-token\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.786230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.785879    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-podinfo\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.789404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.788187    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-envoy\") pod \"e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") " pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.877040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.876955    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:02.953238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:02.952496    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:03.087451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.087426    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263fca38-ec96-41fe-9a14-5346ffd41192" path="/var/lib/kubelet/pods/263fca38-ec96-41fe-9a14-5346ffd41192/volumes"
Nov 03 13:25:03.088429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.088408    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284d1e11-52e5-4c88-925a-35d22f1239d9" path="/var/lib/kubelet/pods/284d1e11-52e5-4c88-925a-35d22f1239d9/volumes"
Nov 03 13:25:03.090262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.090138    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31b93da-47a4-45cf-a17d-836371568cfc" path="/var/lib/kubelet/pods/d31b93da-47a4-45cf-a17d-836371568cfc/volumes"
Nov 03 13:25:03.202991 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:03.202947    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6528d4c3_5e92_4649_8f21_d4ccab65d2a5.slice/crio-c91f2857bf4afa80b5316645944d5585908954b180f53e78b862dbaed3ad802e WatchSource:0}: Error finding container c91f2857bf4afa80b5316645944d5585908954b180f53e78b862dbaed3ad802e: Status 404 returned error can't find the container with id c91f2857bf4afa80b5316645944d5585908954b180f53e78b862dbaed3ad802e
Nov 03 13:25:03.204558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.204534    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"]
Nov 03 13:25:03.205806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.205773    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:03.205884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.205839    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:03.205884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.205868    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:03.466540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.466508    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj" event={"ID":"6528d4c3-5e92-4649-8f21-d4ccab65d2a5","Type":"ContainerStarted","Data":"bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478"}
Nov 03 13:25:03.466736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.466572    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj" event={"ID":"6528d4c3-5e92-4649-8f21-d4ccab65d2a5","Type":"ContainerStarted","Data":"c91f2857bf4afa80b5316645944d5585908954b180f53e78b862dbaed3ad802e"}
Nov 03 13:25:03.491079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.491030    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj" podStartSLOduration=1.491013319 podStartE2EDuration="1.491013319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:03.488147043 +0000 UTC m=+944.565912069" watchObservedRunningTime="2026-11-03 13:25:03.491013319 +0000 UTC m=+944.568778344"
Nov 03 13:25:03.596744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.596703    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:03.878113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:03.878031    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:04.168628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:04.168554    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:04.317664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:04.317629    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:04.613412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:04.613376    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:04.883428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:04.883346    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:05.055370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.055337    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"]
Nov 03 13:25:05.060885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.060855    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7474504f-d7zr8"]
Nov 03 13:25:05.087237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.087175    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e726ad73-dc4c-4265-aa67-30094dd2c4e9" path="/var/lib/kubelet/pods/e726ad73-dc4c-4265-aa67-30094dd2c4e9/volumes"
Nov 03 13:25:05.119709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.119677    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:05.291459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.291428    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:05.455159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.455128    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:05.492829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.492800    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:05.493993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.493970    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:05.656424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.656346    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:05.679089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.679059    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"]
Nov 03 13:25:05.684859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.684837    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:05.696890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.696858    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1a18c8\""
Nov 03 13:25:05.697056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.696919    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1a18c8-dockercfg-pkccj\""
Nov 03 13:25:05.698360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.698334    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"]
Nov 03 13:25:05.743225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.743172    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"]
Nov 03 13:25:05.747833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.747814    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:05.762853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.762824    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"]
Nov 03 13:25:05.785037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.785003    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfd6b\" (UniqueName: \"kubernetes.io/projected/addc9970-fe00-41b6-af12-32f604c2cd74-kube-api-access-hfd6b\") pod \"payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh\" (UID: \"addc9970-fe00-41b6-af12-32f604c2cd74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:05.785204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.785042    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72xqj\" (UniqueName: \"kubernetes.io/projected/40e831ec-f202-437e-a628-be118023bde3-kube-api-access-72xqj\") pod \"payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j\" (UID: \"40e831ec-f202-437e-a628-be118023bde3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:05.785306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.785244    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addc9970-fe00-41b6-af12-32f604c2cd74-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh\" (UID: \"addc9970-fe00-41b6-af12-32f604c2cd74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:05.785498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.785479    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e831ec-f202-437e-a628-be118023bde3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j\" (UID: \"40e831ec-f202-437e-a628-be118023bde3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:05.911537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.908515    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e831ec-f202-437e-a628-be118023bde3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j\" (UID: \"40e831ec-f202-437e-a628-be118023bde3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:05.911537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.908606    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfd6b\" (UniqueName: \"kubernetes.io/projected/addc9970-fe00-41b6-af12-32f604c2cd74-kube-api-access-hfd6b\") pod \"payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh\" (UID: \"addc9970-fe00-41b6-af12-32f604c2cd74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:05.911537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.908635    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72xqj\" (UniqueName: \"kubernetes.io/projected/40e831ec-f202-437e-a628-be118023bde3-kube-api-access-72xqj\") pod \"payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j\" (UID: \"40e831ec-f202-437e-a628-be118023bde3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:05.911537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.908729    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addc9970-fe00-41b6-af12-32f604c2cd74-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh\" (UID: \"addc9970-fe00-41b6-af12-32f604c2cd74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:05.911537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.909749    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addc9970-fe00-41b6-af12-32f604c2cd74-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh\" (UID: \"addc9970-fe00-41b6-af12-32f604c2cd74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:05.911537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.910500    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e831ec-f202-437e-a628-be118023bde3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j\" (UID: \"40e831ec-f202-437e-a628-be118023bde3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:05.963328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.963161    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:05.976231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.969776    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfd6b\" (UniqueName: \"kubernetes.io/projected/addc9970-fe00-41b6-af12-32f604c2cd74-kube-api-access-hfd6b\") pod \"payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh\" (UID: \"addc9970-fe00-41b6-af12-32f604c2cd74\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:05.978831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:05.978701    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72xqj\" (UniqueName: \"kubernetes.io/projected/40e831ec-f202-437e-a628-be118023bde3-kube-api-access-72xqj\") pod \"payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j\" (UID: \"40e831ec-f202-437e-a628-be118023bde3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:06.006026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.005988    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:06.037459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.037424    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"]
Nov 03 13:25:06.046262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.046229    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a4069b-kkp5f"]
Nov 03 13:25:06.065391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.065335    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:06.317308 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:06.317277    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e831ec_f202_437e_a628_be118023bde3.slice/crio-de670f856c9064eeb10f0bef6809a5cb7e494c27b1e65927f0f839982629f0f0 WatchSource:0}: Error finding container de670f856c9064eeb10f0bef6809a5cb7e494c27b1e65927f0f839982629f0f0: Status 404 returned error can't find the container with id de670f856c9064eeb10f0bef6809a5cb7e494c27b1e65927f0f839982629f0f0
Nov 03 13:25:06.320884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.320860    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"]
Nov 03 13:25:06.387330 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:06.387299    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddc9970_fe00_41b6_af12_32f604c2cd74.slice/crio-29d819a3202fb4facd0686c048073e55ab81dd70d32660d9499b6ba1d7f45c0a WatchSource:0}: Error finding container 29d819a3202fb4facd0686c048073e55ab81dd70d32660d9499b6ba1d7f45c0a: Status 404 returned error can't find the container with id 29d819a3202fb4facd0686c048073e55ab81dd70d32660d9499b6ba1d7f45c0a
Nov 03 13:25:06.389508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.389479    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"]
Nov 03 13:25:06.427992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.427837    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:06.506014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.505974    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh" event={"ID":"addc9970-fe00-41b6-af12-32f604c2cd74","Type":"ContainerStarted","Data":"f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6"}
Nov 03 13:25:06.506177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.506041    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh" event={"ID":"addc9970-fe00-41b6-af12-32f604c2cd74","Type":"ContainerStarted","Data":"29d819a3202fb4facd0686c048073e55ab81dd70d32660d9499b6ba1d7f45c0a"}
Nov 03 13:25:06.506177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.506066    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:06.508084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.508055    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j" event={"ID":"40e831ec-f202-437e-a628-be118023bde3","Type":"ContainerStarted","Data":"b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c"}
Nov 03 13:25:06.508237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.508097    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j" event={"ID":"40e831ec-f202-437e-a628-be118023bde3","Type":"ContainerStarted","Data":"de670f856c9064eeb10f0bef6809a5cb7e494c27b1e65927f0f839982629f0f0"}
Nov 03 13:25:06.508237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.508147    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:06.523852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.523805    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh" podStartSLOduration=1.523793371 podStartE2EDuration="1.523793371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:06.522777096 +0000 UTC m=+947.600542125" watchObservedRunningTime="2026-11-03 13:25:06.523793371 +0000 UTC m=+947.601558427"
Nov 03 13:25:06.541537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.541479    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j" podStartSLOduration=1.5414605940000001 podStartE2EDuration="1.541460594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:06.540412225 +0000 UTC m=+947.618177275" watchObservedRunningTime="2026-11-03 13:25:06.541460594 +0000 UTC m=+947.619225622"
Nov 03 13:25:06.747635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.747599    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:06.907373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.899928    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:06.954561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.954222    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:25:06.954561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:06.954290    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:25:07.089244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:07.089158    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97" path="/var/lib/kubelet/pods/ee9b06ea-6c01-40f0-b0ee-0fae07ac7d97/volumes"
Nov 03 13:25:07.289921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:07.289808    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:08.127554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:08.127516    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:08.718195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:08.718160    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"]
Nov 03 13:25:08.841516 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:08.841468    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:08.991010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:08.990937    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:09.236997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:09.236955    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:09.535889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:09.535858    5420 generic.go:356] "Generic (PLEG): container finished" podID="11b73f05-a195-4207-974f-a30c46e31ac1" containerID="eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243" exitCode=0
Nov 03 13:25:09.536108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:09.535943    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" event={"ID":"11b73f05-a195-4207-974f-a30c46e31ac1","Type":"ContainerDied","Data":"eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243"}
Nov 03 13:25:09.536494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:09.536477    5420 scope.go:122] "RemoveContainer" containerID="eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243"
Nov 03 13:25:09.538142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:09.538109    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:09.538284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:09.538199    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:09.538363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:09.538285    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:09.610608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:09.610583    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:10.298171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.298139    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:10.555680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.555592    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" event={"ID":"11b73f05-a195-4207-974f-a30c46e31ac1","Type":"ContainerStarted","Data":"86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2"}
Nov 03 13:25:10.656934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.656901    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:25:10.657148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.657105    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 13:25:10.657235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.657151    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 13:25:10.914335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.914259    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"]
Nov 03 13:25:10.918192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.918176    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:10.945791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.945752    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"distinct-w3-d3ca48-kserve-self-signed-certs\""
Nov 03 13:25:10.970290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.970260    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"]
Nov 03 13:25:10.998636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.998607    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tls-certs\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:10.998782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.998685    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-model-cache\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:10.998782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.998758    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-home\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:10.998897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.998858    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-dshm\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:10.998897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.998882    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wbv\" (UniqueName: \"kubernetes.io/projected/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-kube-api-access-s2wbv\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:10.998981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:10.998966    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tmp-dir\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.051956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.051921    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:11.100485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.100449    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-dshm\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.100684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.100494    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wbv\" (UniqueName: \"kubernetes.io/projected/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-kube-api-access-s2wbv\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.100684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.100571    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tmp-dir\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.100804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.100682    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tls-certs\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.100804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.100756    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-model-cache\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.100804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.100788    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-home\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.101044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.101007    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tmp-dir\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.101184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.101043    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-model-cache\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.101184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.101165    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-home\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.102789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.102771    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-dshm\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.103187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.103166    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tls-certs\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.108467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.108449    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wbv\" (UniqueName: \"kubernetes.io/projected/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-kube-api-access-s2wbv\") pod \"distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.192882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.192797    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:11.269305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.267581    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:11.346232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.345412    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:11.597705 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:11.597672    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a0fafd5_a802_41e7_b2b2_02a11ebf6217.slice/crio-6cdfc892af9a76121f926f7a24359a172c5f4cdb422583a40c3a00d47079b549 WatchSource:0}: Error finding container 6cdfc892af9a76121f926f7a24359a172c5f4cdb422583a40c3a00d47079b549: Status 404 returned error can't find the container with id 6cdfc892af9a76121f926f7a24359a172c5f4cdb422583a40c3a00d47079b549
Nov 03 13:25:11.599190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.599166    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"]
Nov 03 13:25:11.658243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.657065    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 13:25:11.658243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.657122    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 13:25:11.864100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:11.863987    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:12.575028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:12.574993    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w" event={"ID":"9a0fafd5-a802-41e7-b2b2-02a11ebf6217","Type":"ContainerStarted","Data":"7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916"}
Nov 03 13:25:12.575623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:12.575036    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w" event={"ID":"9a0fafd5-a802-41e7-b2b2-02a11ebf6217","Type":"ContainerStarted","Data":"6cdfc892af9a76121f926f7a24359a172c5f4cdb422583a40c3a00d47079b549"}
Nov 03 13:25:12.600829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:12.600768    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w" podStartSLOduration=2.60074969 podStartE2EDuration="2.60074969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:12.598354441 +0000 UTC m=+953.676119466" watchObservedRunningTime="2026-11-03 13:25:12.60074969 +0000 UTC m=+953.678514718"
Nov 03 13:25:12.657808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:12.657773    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 13:25:12.657980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:12.657826    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 13:25:12.681197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:12.681163    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:12.817884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:12.817852    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:12.961550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:12.961522    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:13.275398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:13.275306    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:13.442237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:13.441659    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:13.657240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:13.657133    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 13:25:13.657240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:13.657189    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 13:25:13.718882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:13.718842    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc" podUID="03557d4b-676e-4fe8-88a1-5695df46ccc9" containerName="payload-processing" containerID="cri-o://195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d" gracePeriod=25
Nov 03 13:25:14.055715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.055674    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"]
Nov 03 13:25:14.062329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.062305    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3f5c81-2qpvn"]
Nov 03 13:25:14.064430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.064136    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:25:14.108873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.108844    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:14.171045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.170933    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03557d4b-676e-4fe8-88a1-5695df46ccc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03557d4b-676e-4fe8-88a1-5695df46ccc9-plugins-config-volume\") pod \"03557d4b-676e-4fe8-88a1-5695df46ccc9\" (UID: \"03557d4b-676e-4fe8-88a1-5695df46ccc9\") "
Nov 03 13:25:14.171362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.171339    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03557d4b-676e-4fe8-88a1-5695df46ccc9-kube-api-access-lx9mj\" (UniqueName: \"kubernetes.io/projected/03557d4b-676e-4fe8-88a1-5695df46ccc9-kube-api-access-lx9mj\") pod \"03557d4b-676e-4fe8-88a1-5695df46ccc9\" (UID: \"03557d4b-676e-4fe8-88a1-5695df46ccc9\") "
Nov 03 13:25:14.171598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.171341    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03557d4b-676e-4fe8-88a1-5695df46ccc9-plugins-config-volume" pod "03557d4b-676e-4fe8-88a1-5695df46ccc9" (UID: "03557d4b-676e-4fe8-88a1-5695df46ccc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:14.172636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.172610    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03557d4b-676e-4fe8-88a1-5695df46ccc9-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:14.173360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.173341    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03557d4b-676e-4fe8-88a1-5695df46ccc9-kube-api-access-lx9mj" pod "03557d4b-676e-4fe8-88a1-5695df46ccc9" (UID: "03557d4b-676e-4fe8-88a1-5695df46ccc9"). InnerVolumeSpecName "kube-api-access-lx9mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:14.274063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.274024    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9mj\" (UniqueName: \"kubernetes.io/projected/03557d4b-676e-4fe8-88a1-5695df46ccc9-kube-api-access-lx9mj\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:14.490877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.490843    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:25:14.601249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.601194    5420 generic.go:356] "Generic (PLEG): container finished" podID="03557d4b-676e-4fe8-88a1-5695df46ccc9" containerID="195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d" exitCode=0
Nov 03 13:25:14.602862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.602835    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"
Nov 03 13:25:14.603629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.603606    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc" event={"ID":"03557d4b-676e-4fe8-88a1-5695df46ccc9","Type":"ContainerDied","Data":"195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d"}
Nov 03 13:25:14.603744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.603652    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc" event={"ID":"03557d4b-676e-4fe8-88a1-5695df46ccc9","Type":"ContainerDied","Data":"80ce2bd8c65081bcf61b8811504acf46b2822ca53fa5a5fce7af96e7a88fed61"}
Nov 03 13:25:14.603744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.603675    5420 scope.go:122] "RemoveContainer" containerID="195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d"
Nov 03 13:25:14.619101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.619085    5420 scope.go:122] "RemoveContainer" containerID="195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d"
Nov 03 13:25:14.619400 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:25:14.619378    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d\": container with ID starting with 195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d not found: ID does not exist" containerID="195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d"
Nov 03 13:25:14.619498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.619406    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d"} err="failed to get container status \"195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d\": rpc error: code = NotFound desc = could not find container \"195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d\": container with ID starting with 195c5266bbaff503da7c7e2901d1988ef938b9d99e377d8e7ea66c607dbecc8d not found: ID does not exist"
Nov 03 13:25:14.651449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.651403    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"]
Nov 03 13:25:14.656558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.656537    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b26c22-c85b5df6c-j5vvc"]
Nov 03 13:25:14.657183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.657165    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 13:25:14.657263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.657244    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 13:25:14.685005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.684978    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:14.933710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:14.929342    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:15.086630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:15.086600    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03557d4b-676e-4fe8-88a1-5695df46ccc9" path="/var/lib/kubelet/pods/03557d4b-676e-4fe8-88a1-5695df46ccc9/volumes"
Nov 03 13:25:15.087105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:15.087090    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3369c1-1168-45e1-838c-5e8e9645dd3c" path="/var/lib/kubelet/pods/5e3369c1-1168-45e1-838c-5e8e9645dd3c/volumes"
Nov 03 13:25:15.609237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:15.609164    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:15.668453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:15.668420    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 13:25:15.670158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:15.670126    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 13:25:15.768704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:15.768667    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:15.914018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:15.913940    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:16.092154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:16.092124    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:16.418533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:16.418502    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"]
Nov 03 13:25:16.418774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:16.418750    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy" containerID="cri-o://20030bf113a317db59c01543ccfd9f43fc8c75fd57845f66ef3a6b110d11c764" gracePeriod=30
Nov 03 13:25:16.485569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:16.485535    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:16.662299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:16.662270    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:25:16.662838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:16.662813    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:25:16.663675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:16.663658    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:25:16.912135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:16.912099    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:17.303374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:17.303337    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:17.457258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:17.457200    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:17.520980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:17.520952    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:25:17.521178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:17.521041    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:25:17.615312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:17.615227    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:17.847005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:17.846972    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:18.032042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:18.032001    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"]
Nov 03 13:25:18.037629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:18.037600    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0506a5e3-dgsx5"]
Nov 03 13:25:18.128607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:18.128567    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:18.426718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:18.426635    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:19.023765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:19.023739    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"]
Nov 03 13:25:19.028748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:19.028726    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-259e5448-p5xzk"]
Nov 03 13:25:19.085291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:19.085262    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c449edd-6bfc-4903-80b3-1b0c09b1d158" path="/var/lib/kubelet/pods/9c449edd-6bfc-4903-80b3-1b0c09b1d158/volumes"
Nov 03 13:25:19.085681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:19.085667    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69242a8-f26c-421b-a78c-cae43aaf4f25" path="/var/lib/kubelet/pods/b69242a8-f26c-421b-a78c-cae43aaf4f25/volumes"
Nov 03 13:25:19.095709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:19.095684    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:19.896817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:19.896786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:20.080039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:20.080009    5420 scope.go:122] "RemoveContainer" containerID="98a2a7f1bd2c8404ba7c57f7d00e44788802cf74c363cad72f015c5e7a088c69"
Nov 03 13:25:20.574231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:20.574179    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:21.031370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.031333    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"]
Nov 03 13:25:21.032027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.032012    5420 cpu_manager.go:414] "containerMap: removing container" podUID="03557d4b-676e-4fe8-88a1-5695df46ccc9" containerName="payload-processing"
Nov 03 13:25:21.032077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.032028    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03557d4b-676e-4fe8-88a1-5695df46ccc9" containerName="payload-processing"
Nov 03 13:25:21.032150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.032141    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="03557d4b-676e-4fe8-88a1-5695df46ccc9" containerName="payload-processing"
Nov 03 13:25:21.037937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.037921    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.043959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.043936    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d3ca48-openshift-default-dockercfg-lgz27\""
Nov 03 13:25:21.050609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.050587    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"]
Nov 03 13:25:21.204243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.204186    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:21.226764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.226724    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.226940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.226918    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.227003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.226956    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.227206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.227181    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.227333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.227289    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.227397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.227332    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0c07ab-fa98-462e-8fbc-6119172109cb-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.227454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.227437    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42js5\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-kube-api-access-42js5\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.227493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.227464    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.227728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.227716    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.268754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.268716    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:21.281206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.281180    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:21.329502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329425    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.329502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329462    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.329502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329488    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0c07ab-fa98-462e-8fbc-6119172109cb-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.329808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329540    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42js5\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-kube-api-access-42js5\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.329808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329560    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.329808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329588    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.329808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329613    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.330025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329857    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.330025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329912    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.330025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.329988    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.330025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.330007    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.330335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.330083    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.330335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.330324    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0c07ab-fa98-462e-8fbc-6119172109cb-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.330414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.330385    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.332049 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.332031    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.332460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.332442    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.338788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.338766    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42js5\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-kube-api-access-42js5\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.338938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.338917    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.354313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.354288    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:21.666757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.665545    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:21.666757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.665633    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:21.666757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.665678    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:21.673052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.673028    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"]
Nov 03 13:25:21.695346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.695314    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:21.697695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.697340    5420 generic.go:356] "Generic (PLEG): container finished" podID="75bdcc44-6914-4774-9b98-2f77caa51434" containerID="20030bf113a317db59c01543ccfd9f43fc8c75fd57845f66ef3a6b110d11c764" exitCode=0
Nov 03 13:25:21.697695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.697516    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" event={"ID":"75bdcc44-6914-4774-9b98-2f77caa51434","Type":"ContainerDied","Data":"20030bf113a317db59c01543ccfd9f43fc8c75fd57845f66ef3a6b110d11c764"}
Nov 03 13:25:21.697695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.697579    5420 scope.go:122] "RemoveContainer" containerID="7d27bffe5874fb86ddbf72c3926b9f3dec917c210d09394ecc88511098123ebb"
Nov 03 13:25:21.702809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.702737    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" event={"ID":"7e0c07ab-fa98-462e-8fbc-6119172109cb","Type":"ContainerStarted","Data":"c5bd191c19de735d643cc8a00f1ffb415e6871bc3486634833995c708e8af45e"}
Nov 03 13:25:21.720077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.719729    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:25:21.749473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.749449    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:25:21.759233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759198    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75bdcc44-6914-4774-9b98-2f77caa51434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bdcc44-6914-4774-9b98-2f77caa51434-istiod-ca-cert\") pod \"75bdcc44-6914-4774-9b98-2f77caa51434\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") "
Nov 03 13:25:21.759343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759268    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-socket\") pod \"75bdcc44-6914-4774-9b98-2f77caa51434\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") "
Nov 03 13:25:21.759343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759326    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-certs\") pod \"75bdcc44-6914-4774-9b98-2f77caa51434\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") "
Nov 03 13:25:21.759451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759379    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-envoy\") pod \"75bdcc44-6914-4774-9b98-2f77caa51434\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") "
Nov 03 13:25:21.759492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759446    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-credential-socket\") pod \"75bdcc44-6914-4774-9b98-2f77caa51434\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") "
Nov 03 13:25:21.759528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759500    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-istio-token\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-istio-token\") pod \"75bdcc44-6914-4774-9b98-2f77caa51434\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") "
Nov 03 13:25:21.759594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759554    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-socket" pod "75bdcc44-6914-4774-9b98-2f77caa51434" (UID: "75bdcc44-6914-4774-9b98-2f77caa51434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:21.759642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759619    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-certs" pod "75bdcc44-6914-4774-9b98-2f77caa51434" (UID: "75bdcc44-6914-4774-9b98-2f77caa51434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:21.759686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759658    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-data\") pod \"75bdcc44-6914-4774-9b98-2f77caa51434\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") "
Nov 03 13:25:21.759742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759720    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-kube-api-access-hsxpd\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-kube-api-access-hsxpd\") pod \"75bdcc44-6914-4774-9b98-2f77caa51434\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") "
Nov 03 13:25:21.759816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759746    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-credential-socket" pod "75bdcc44-6914-4774-9b98-2f77caa51434" (UID: "75bdcc44-6914-4774-9b98-2f77caa51434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:21.759816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759758    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75bdcc44-6914-4774-9b98-2f77caa51434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bdcc44-6914-4774-9b98-2f77caa51434-istio-podinfo\") pod \"75bdcc44-6914-4774-9b98-2f77caa51434\" (UID: \"75bdcc44-6914-4774-9b98-2f77caa51434\") "
Nov 03 13:25:21.760347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.759949    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-data" pod "75bdcc44-6914-4774-9b98-2f77caa51434" (UID: "75bdcc44-6914-4774-9b98-2f77caa51434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:21.760505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.760414    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:21.760505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.760436    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:21.760505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.760455    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:21.760505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.760473    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:21.761988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.761962    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75bdcc44-6914-4774-9b98-2f77caa51434-istiod-ca-cert" pod "75bdcc44-6914-4774-9b98-2f77caa51434" (UID: "75bdcc44-6914-4774-9b98-2f77caa51434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:21.762761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.762705    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-envoy" pod "75bdcc44-6914-4774-9b98-2f77caa51434" (UID: "75bdcc44-6914-4774-9b98-2f77caa51434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:21.763906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.762825    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75bdcc44-6914-4774-9b98-2f77caa51434-istio-podinfo" pod "75bdcc44-6914-4774-9b98-2f77caa51434" (UID: "75bdcc44-6914-4774-9b98-2f77caa51434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:25:21.764582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.764557    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-kube-api-access-hsxpd" pod "75bdcc44-6914-4774-9b98-2f77caa51434" (UID: "75bdcc44-6914-4774-9b98-2f77caa51434"). InnerVolumeSpecName "kube-api-access-hsxpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:21.764808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.764772    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-istio-token" pod "75bdcc44-6914-4774-9b98-2f77caa51434" (UID: "75bdcc44-6914-4774-9b98-2f77caa51434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:21.862142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.862060    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bdcc44-6914-4774-9b98-2f77caa51434-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:21.862401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.862379    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:21.862530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.862513    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsxpd\" (UniqueName: \"kubernetes.io/projected/75bdcc44-6914-4774-9b98-2f77caa51434-kube-api-access-hsxpd\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:21.862632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.862619    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bdcc44-6914-4774-9b98-2f77caa51434-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:21.862752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.862738    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bdcc44-6914-4774-9b98-2f77caa51434-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:21.894250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:21.894176    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:22.099093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:22.099053    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:22.712797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:22.712765    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"
Nov 03 13:25:22.713290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:22.712791    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55" event={"ID":"75bdcc44-6914-4774-9b98-2f77caa51434","Type":"ContainerDied","Data":"80734a1475a73f2cdb2b2a8973ceccd921dac8afeb0f9508888af7e25111aa50"}
Nov 03 13:25:22.713290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:22.712856    5420 scope.go:122] "RemoveContainer" containerID="20030bf113a317db59c01543ccfd9f43fc8c75fd57845f66ef3a6b110d11c764"
Nov 03 13:25:22.715427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:22.715400    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" event={"ID":"7e0c07ab-fa98-462e-8fbc-6119172109cb","Type":"ContainerStarted","Data":"f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c"}
Nov 03 13:25:22.762022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:22.761969    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" podStartSLOduration=1.761956086 podStartE2EDuration="1.761956086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:22.759073067 +0000 UTC m=+963.836838095" watchObservedRunningTime="2026-11-03 13:25:22.761956086 +0000 UTC m=+963.839721111"
Nov 03 13:25:22.780429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:22.780326    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"]
Nov 03 13:25:22.785593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:22.785564    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b26c22-openshift-default-5d5cff7997-fzr55"]
Nov 03 13:25:22.802326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:22.802300    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:23.088393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.088176    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" path="/var/lib/kubelet/pods/75bdcc44-6914-4774-9b98-2f77caa51434/volumes"
Nov 03 13:25:23.169781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.169750    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:23.182814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.182788    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"]
Nov 03 13:25:23.183413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.183400    5420 cpu_manager.go:414] "containerMap: removing container" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy"
Nov 03 13:25:23.183470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.183414    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy"
Nov 03 13:25:23.183470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.183458    5420 cpu_manager.go:414] "containerMap: removing container" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy"
Nov 03 13:25:23.183470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.183464    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy"
Nov 03 13:25:23.183581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.183572    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy"
Nov 03 13:25:23.183623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.183585    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bdcc44-6914-4774-9b98-2f77caa51434" containerName="istio-proxy"
Nov 03 13:25:23.188666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.188635    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:23.195200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.195175    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"]
Nov 03 13:25:23.196372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.196351    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-8f8aa8\""
Nov 03 13:25:23.197342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.197322    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-8f8aa8-dockercfg-wzt82\""
Nov 03 13:25:23.253688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.253649    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"]
Nov 03 13:25:23.259156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.259120    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:23.267102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.267076    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"]
Nov 03 13:25:23.282478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.282445    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5cjq\" (UniqueName: \"kubernetes.io/projected/b788ffb7-d4b9-4ed0-a537-48ab5f115814-kube-api-access-s5cjq\") pod \"payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd\" (UID: \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:23.282642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.282587    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lkk\" (UniqueName: \"kubernetes.io/projected/7149f288-00a3-40a7-9761-590f94057b00-kube-api-access-t4lkk\") pod \"payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp\" (UID: \"7149f288-00a3-40a7-9761-590f94057b00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:23.282867 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.282845    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b788ffb7-d4b9-4ed0-a537-48ab5f115814-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd\" (UID: \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:23.282952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.282935    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149f288-00a3-40a7-9761-590f94057b00-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp\" (UID: \"7149f288-00a3-40a7-9761-590f94057b00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:23.390291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.365677    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:23.407230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.394922    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:25:23.425921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.395464    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:23.446179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.427508    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b788ffb7-d4b9-4ed0-a537-48ab5f115814-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd\" (UID: \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:23.446179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.427861    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149f288-00a3-40a7-9761-590f94057b00-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp\" (UID: \"7149f288-00a3-40a7-9761-590f94057b00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:23.446179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.428018    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5cjq\" (UniqueName: \"kubernetes.io/projected/b788ffb7-d4b9-4ed0-a537-48ab5f115814-kube-api-access-s5cjq\") pod \"payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd\" (UID: \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:23.446179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.430580    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149f288-00a3-40a7-9761-590f94057b00-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp\" (UID: \"7149f288-00a3-40a7-9761-590f94057b00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:23.460292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.452631    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lkk\" (UniqueName: \"kubernetes.io/projected/7149f288-00a3-40a7-9761-590f94057b00-kube-api-access-t4lkk\") pod \"payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp\" (UID: \"7149f288-00a3-40a7-9761-590f94057b00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:23.461437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.461382    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 13:25:23.481455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.481356    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b788ffb7-d4b9-4ed0-a537-48ab5f115814-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd\" (UID: \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:23.484301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.484274    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lkk\" (UniqueName: \"kubernetes.io/projected/7149f288-00a3-40a7-9761-590f94057b00-kube-api-access-t4lkk\") pod \"payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp\" (UID: \"7149f288-00a3-40a7-9761-590f94057b00\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:23.486290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.486237    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5cjq\" (UniqueName: \"kubernetes.io/projected/b788ffb7-d4b9-4ed0-a537-48ab5f115814-kube-api-access-s5cjq\") pod \"payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd\" (UID: \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:23.512765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.512736    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:23.576734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.576697    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:23.616623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.616518    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:23.858369 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:23.858338    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb788ffb7_d4b9_4ed0_a537_48ab5f115814.slice/crio-e815ae99071ad9dc1d815eaa183ce878f55287f1ff5ea09c8b8e3206085dbafd WatchSource:0}: Error finding container e815ae99071ad9dc1d815eaa183ce878f55287f1ff5ea09c8b8e3206085dbafd: Status 404 returned error can't find the container with id e815ae99071ad9dc1d815eaa183ce878f55287f1ff5ea09c8b8e3206085dbafd
Nov 03 13:25:23.871157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.859832    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"]
Nov 03 13:25:23.952854 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:23.952829    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7149f288_00a3_40a7_9761_590f94057b00.slice/crio-39bb0cefa84c9efcf5d7614e53c5fcc6c7c7ac6dc8d62016953e1887244b870f WatchSource:0}: Error finding container 39bb0cefa84c9efcf5d7614e53c5fcc6c7c7ac6dc8d62016953e1887244b870f: Status 404 returned error can't find the container with id 39bb0cefa84c9efcf5d7614e53c5fcc6c7c7ac6dc8d62016953e1887244b870f
Nov 03 13:25:23.954170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:23.954145    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"]
Nov 03 13:25:24.283833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.283801    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:24.365747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.365713    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:25:24.365957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.365769    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 13:25:24.548659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.548575    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:24.752254 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:25:24.752198    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:25:24.774543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.774511    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd" event={"ID":"b788ffb7-d4b9-4ed0-a537-48ab5f115814","Type":"ContainerStarted","Data":"6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43"}
Nov 03 13:25:24.774724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.774568    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd" event={"ID":"b788ffb7-d4b9-4ed0-a537-48ab5f115814","Type":"ContainerStarted","Data":"e815ae99071ad9dc1d815eaa183ce878f55287f1ff5ea09c8b8e3206085dbafd"}
Nov 03 13:25:24.774724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.774597    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:24.776319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.776291    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp" event={"ID":"7149f288-00a3-40a7-9761-590f94057b00","Type":"ContainerStarted","Data":"3f8c4d29ab1ba7255b17cbf4d974bd461113acc4021a7738e1a4a24301096533"}
Nov 03 13:25:24.776447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.776338    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp" event={"ID":"7149f288-00a3-40a7-9761-590f94057b00","Type":"ContainerStarted","Data":"39bb0cefa84c9efcf5d7614e53c5fcc6c7c7ac6dc8d62016953e1887244b870f"}
Nov 03 13:25:24.795379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.795330    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd" podStartSLOduration=1.795318094 podStartE2EDuration="1.795318094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:24.791896725 +0000 UTC m=+965.869661753" watchObservedRunningTime="2026-11-03 13:25:24.795318094 +0000 UTC m=+965.873083120"
Nov 03 13:25:24.850400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.850279    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp" podStartSLOduration=1.850257037 podStartE2EDuration="1.850257037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:24.811584501 +0000 UTC m=+965.889349526" watchObservedRunningTime="2026-11-03 13:25:24.850257037 +0000 UTC m=+965.928022068"
Nov 03 13:25:24.961038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:24.961003    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:25.355561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.355530    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:25:25.355764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.355583    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 13:25:25.565857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.565819    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:25.732263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.726746    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:25.787234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.786238    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:25.874095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.874064    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:25.980063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.980032    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"]
Nov 03 13:25:25.985350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.985287    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"]
Nov 03 13:25:25.985471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.985374    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:25.990943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.990916    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:25:25.991056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.990984    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:25:25.991056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.991011    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:25:25.991271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:25.991190    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:25:26.104491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.104456    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzhv\" (UniqueName: \"kubernetes.io/projected/316b29df-f26e-47e4-b90f-391d5fb7b7fc-kube-api-access-lnzhv\") pod \"maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.104665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.104595    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.104665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.104633    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.205977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.205941    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzhv\" (UniqueName: \"kubernetes.io/projected/316b29df-f26e-47e4-b90f-391d5fb7b7fc-kube-api-access-lnzhv\") pod \"maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.206162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.206061    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.206162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.206089    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.206769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.206747    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.208926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.208898    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.214390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.214363    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:26.217338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.217304    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzhv\" (UniqueName: \"kubernetes.io/projected/316b29df-f26e-47e4-b90f-391d5fb7b7fc-kube-api-access-lnzhv\") pod \"maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.301807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.301715    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:26.355087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.355057    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:25:26.355318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.355107    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 13:25:26.504029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.503940    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:26.619881 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:26.619842    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316b29df_f26e_47e4_b90f_391d5fb7b7fc.slice/crio-5cba82a262b44849bc80ccda03252780edecfddfcbd0838d65f51b40da88f06a WatchSource:0}: Error finding container 5cba82a262b44849bc80ccda03252780edecfddfcbd0838d65f51b40da88f06a: Status 404 returned error can't find the container with id 5cba82a262b44849bc80ccda03252780edecfddfcbd0838d65f51b40da88f06a
Nov 03 13:25:26.635408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.622576    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"]
Nov 03 13:25:26.795855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.795819    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg" event={"ID":"316b29df-f26e-47e4-b90f-391d5fb7b7fc","Type":"ContainerStarted","Data":"5cba82a262b44849bc80ccda03252780edecfddfcbd0838d65f51b40da88f06a"}
Nov 03 13:25:26.847253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.846517    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:26.998357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:26.998325    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"]
Nov 03 13:25:27.004406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.004382    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.012845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.012823    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"distinct-2-w3-d3ca48-kserve-self-signed-certs\""
Nov 03 13:25:27.017854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.017824    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"]
Nov 03 13:25:27.056654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.056222    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:27.119706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.119663    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tmp-dir\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.119881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.119716    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tls-certs\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.119881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.119831    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-model-cache\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.119982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.119887    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7s59\" (UniqueName: \"kubernetes.io/projected/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-kube-api-access-b7s59\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.120037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.120006    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-dshm\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.120207 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.120190    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-home\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.219868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.219838    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:27.232233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.229119    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-home\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.232233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.229564    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-home\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.232233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.229713    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tmp-dir\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.232233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.230000    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tmp-dir\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.232233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.230061    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tls-certs\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.232233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.230561    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-model-cache\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.232233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.230625    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7s59\" (UniqueName: \"kubernetes.io/projected/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-kube-api-access-b7s59\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.232233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.230685    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-dshm\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.232233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.231391    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-model-cache\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.255272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.253642    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-dshm\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.255272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.254958    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tls-certs\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.255552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.255532    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7s59\" (UniqueName: \"kubernetes.io/projected/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-kube-api-access-b7s59\") pod \"distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") " pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.322236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.319072    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:27.359648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.359562    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:25:27.359648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.359612    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 13:25:27.366672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.366642    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:27.715609 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:27.715583    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5fa55b7_f94d_495d_b8ca_8c7b97632cb6.slice/crio-7a8d135c3230e341cd17147bd9a8bdb5518abf6444bf1e4748948e9a30eab859 WatchSource:0}: Error finding container 7a8d135c3230e341cd17147bd9a8bdb5518abf6444bf1e4748948e9a30eab859: Status 404 returned error can't find the container with id 7a8d135c3230e341cd17147bd9a8bdb5518abf6444bf1e4748948e9a30eab859
Nov 03 13:25:27.718732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.718678    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"]
Nov 03 13:25:27.803247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.796127    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:27.867804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.867752    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv" event={"ID":"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6","Type":"ContainerStarted","Data":"7a8d135c3230e341cd17147bd9a8bdb5518abf6444bf1e4748948e9a30eab859"}
Nov 03 13:25:27.900029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.899999    5420 generic.go:356] "Generic (PLEG): container finished" podID="316b29df-f26e-47e4-b90f-391d5fb7b7fc" containerID="8d6666b8baef02009d24d513153904fcc6c3233b7a8b561e7317603bb341d81a" exitCode=0
Nov 03 13:25:27.900487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.900460    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg" event={"ID":"316b29df-f26e-47e4-b90f-391d5fb7b7fc","Type":"ContainerDied","Data":"8d6666b8baef02009d24d513153904fcc6c3233b7a8b561e7317603bb341d81a"}
Nov 03 13:25:27.949656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:27.949627    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:28.362640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.362607    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 13:25:28.363267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.363196    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 13:25:28.608872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.608667    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:28.754232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.750761    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:28.887231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.887125    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"]
Nov 03 13:25:28.894883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.894858    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:28.900268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.900241    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-4d65870da959\""
Nov 03 13:25:28.900407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.900355    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"openshift-service-ca.crt\""
Nov 03 13:25:28.901075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.901051    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"routing-overlay\""
Nov 03 13:25:28.901228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.901137    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ef1693\""
Nov 03 13:25:28.901408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.901387    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-e2e--393fd9ed-dockercfg-fnc52\""
Nov 03 13:25:28.901487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.901431    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"kube-root-ca.crt\""
Nov 03 13:25:28.905177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.904921    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"]
Nov 03 13:25:28.979631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.978999    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv" event={"ID":"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6","Type":"ContainerStarted","Data":"41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79"}
Nov 03 13:25:28.991271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.991239    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tts2z\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-kube-api-access-tts2z\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:28.991594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.991565    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:28.991714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.991670    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:28.991775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:28.991710    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.007673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.007615    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv" podStartSLOduration=3.007594942 podStartE2EDuration="3.007594942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:28.999057054 +0000 UTC m=+970.076822081" watchObservedRunningTime="2026-11-03 13:25:29.007594942 +0000 UTC m=+970.085359968"
Nov 03 13:25:29.070233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.069278    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"]
Nov 03 13:25:29.105231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.104561    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts2z\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-kube-api-access-tts2z\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.105231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.104644    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.105231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.104752    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.105231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.104789    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.136829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.118609    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.151433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.138142    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tts2z\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-kube-api-access-tts2z\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.151433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.138556    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.159637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.142627    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.172289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.172257    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"]
Nov 03 13:25:29.203237 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:25:29.193406    5420 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-f0a895c6-z9bkw\" is forbidden: User \"system:node:ip-10-0-139-248.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-248.us-east-2.compute.internal' and this object" podUID="4a894dde-efd1-488a-937b-5f69b9905c9d" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a895c6-z9bkw"
Nov 03 13:25:29.299890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.299573    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:29.365758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.365727    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:29.414140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.409487    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:29.579767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.579734    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:29.680300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.673813    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-xks-ca\") pod \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") "
Nov 03 13:25:29.680300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.673875    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316b29df-f26e-47e4-b90f-391d5fb7b7fc-kube-api-access-lnzhv\" (UniqueName: \"kubernetes.io/projected/316b29df-f26e-47e4-b90f-391d5fb7b7fc-kube-api-access-lnzhv\") pod \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") "
Nov 03 13:25:29.680300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.674121    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-openshift-ca\") pod \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\" (UID: \"316b29df-f26e-47e4-b90f-391d5fb7b7fc\") "
Nov 03 13:25:29.680300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.674964    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-openshift-ca" pod "316b29df-f26e-47e4-b90f-391d5fb7b7fc" (UID: "316b29df-f26e-47e4-b90f-391d5fb7b7fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:29.680300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.677516    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-xks-ca" pod "316b29df-f26e-47e4-b90f-391d5fb7b7fc" (UID: "316b29df-f26e-47e4-b90f-391d5fb7b7fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:25:29.690597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.690570    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316b29df-f26e-47e4-b90f-391d5fb7b7fc-kube-api-access-lnzhv" pod "316b29df-f26e-47e4-b90f-391d5fb7b7fc" (UID: "316b29df-f26e-47e4-b90f-391d5fb7b7fc"). InnerVolumeSpecName "kube-api-access-lnzhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:29.776418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.776385    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:29.776546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.776421    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzhv\" (UniqueName: \"kubernetes.io/projected/316b29df-f26e-47e4-b90f-391d5fb7b7fc-kube-api-access-lnzhv\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:29.776546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.776438    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/316b29df-f26e-47e4-b90f-391d5fb7b7fc-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:29.790942 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:29.790916    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de9561a_b039_49f1_a901_d338e7e49820.slice/crio-39724c5963f9b6d7d51673857f6f66f8b4612921c9237df5d493a77b264dc95d WatchSource:0}: Error finding container 39724c5963f9b6d7d51673857f6f66f8b4612921c9237df5d493a77b264dc95d: Status 404 returned error can't find the container with id 39724c5963f9b6d7d51673857f6f66f8b4612921c9237df5d493a77b264dc95d
Nov 03 13:25:29.794826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.794800    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"]
Nov 03 13:25:29.997241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.996879    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh" event={"ID":"0de9561a-b039-49f1-a901-d338e7e49820","Type":"ContainerStarted","Data":"aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28"}
Nov 03 13:25:29.997241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.996945    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh" event={"ID":"0de9561a-b039-49f1-a901-d338e7e49820","Type":"ContainerStarted","Data":"39724c5963f9b6d7d51673857f6f66f8b4612921c9237df5d493a77b264dc95d"}
Nov 03 13:25:29.998462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:29.998408    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:30.001857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.000953    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"
Nov 03 13:25:30.002285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.002155    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg" event={"ID":"316b29df-f26e-47e4-b90f-391d5fb7b7fc","Type":"ContainerDied","Data":"5cba82a262b44849bc80ccda03252780edecfddfcbd0838d65f51b40da88f06a"}
Nov 03 13:25:30.002579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.002400    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cba82a262b44849bc80ccda03252780edecfddfcbd0838d65f51b40da88f06a"
Nov 03 13:25:30.061260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.060829    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh" podStartSLOduration=2.060808863 podStartE2EDuration="2.060808863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:30.05962882 +0000 UTC m=+971.137393871" watchObservedRunningTime="2026-11-03 13:25:30.060808863 +0000 UTC m=+971.138573890"
Nov 03 13:25:30.106827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.106799    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:30.271540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.271455    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:30.289474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.289442    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:30.289978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.289959    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:30.290839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.290819    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:25:30.545818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.545729    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:30.955570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:30.955533    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:31.086760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:31.086725    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a894dde-efd1-488a-937b-5f69b9905c9d" path="/var/lib/kubelet/pods/4a894dde-efd1-488a-937b-5f69b9905c9d/volumes"
Nov 03 13:25:31.508819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:31.508775    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:32.050318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.050279    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"]
Nov 03 13:25:32.055598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.055570    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29189215-57mr9"]
Nov 03 13:25:32.109124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.109049    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:32.169169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.169137    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"]
Nov 03 13:25:32.170639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.170206    5420 cpu_manager.go:414] "containerMap: removing container" podUID="316b29df-f26e-47e4-b90f-391d5fb7b7fc" containerName="revoke-keys"
Nov 03 13:25:32.170639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.170252    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316b29df-f26e-47e4-b90f-391d5fb7b7fc" containerName="revoke-keys"
Nov 03 13:25:32.170639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.170430    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="316b29df-f26e-47e4-b90f-391d5fb7b7fc" containerName="revoke-keys"
Nov 03 13:25:32.175113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.175096    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.182299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.181507    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 13:25:32.182299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.181507    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 13:25:32.182299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.181631    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 13:25:32.182299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.182048    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8strr\""
Nov 03 13:25:32.184422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.184397    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"]
Nov 03 13:25:32.321284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.320428    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.321284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.320530    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62sl\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-kube-api-access-d62sl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.321284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.320605    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.321284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.320676    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.321284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.320712    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf961da-3329-4396-96f5-817279fd5f03-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.321284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.320837    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.321284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.320900    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.445237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.443181    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.445237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.443283    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.445237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.443322    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.445237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.443419    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62sl\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-kube-api-access-d62sl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.445237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.443499    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.445237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.443575    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.445237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.443611    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf961da-3329-4396-96f5-817279fd5f03-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.446056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.446023    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.447261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.446607    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf961da-3329-4396-96f5-817279fd5f03-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.448344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.448324    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.448433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.448415    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.448633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.448617    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.453899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.453876    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.454140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.454123    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62sl\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-kube-api-access-d62sl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dvwqs\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.492808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.492785    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:32.786427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.786396    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:32.809178 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:32.809144    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf961da_3329_4396_96f5_817279fd5f03.slice/crio-5a76b968e6211910c1da441d222d13f6258eae4e799afba85fca89a603bfcad1 WatchSource:0}: Error finding container 5a76b968e6211910c1da441d222d13f6258eae4e799afba85fca89a603bfcad1: Status 404 returned error can't find the container with id 5a76b968e6211910c1da441d222d13f6258eae4e799afba85fca89a603bfcad1
Nov 03 13:25:32.810396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.810374    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"]
Nov 03 13:25:32.811358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.811341    5420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:25:32.950075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:32.949981    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:33.061456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:33.061414    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs" event={"ID":"7cf961da-3329-4396-96f5-817279fd5f03","Type":"ContainerStarted","Data":"5a76b968e6211910c1da441d222d13f6258eae4e799afba85fca89a603bfcad1"}
Nov 03 13:25:33.086333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:33.086300    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedd069f-adbf-4c74-94d8-ac5782e5e1c2" path="/var/lib/kubelet/pods/bedd069f-adbf-4c74-94d8-ac5782e5e1c2/volumes"
Nov 03 13:25:33.087648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:33.087617    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:33.816901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:33.816858    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:34.031191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:34.031144    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:34.499165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:34.497944    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:35.196687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:35.196656    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:35.635084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:35.635032    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"]
Nov 03 13:25:35.789817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:35.789791    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:25:35.976700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:35.976668    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:36.023568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:36.023534    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:36.023674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:36.023598    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:25:36.273876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:36.273802    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:36.411526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:36.411496    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:36.558514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:36.537390    5420 patch_prober.go:29] interesting pod/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:25:36.558514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:36.537497    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj" podUID="6528d4c3-5e92-4649-8f21-d4ccab65d2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:25:36.711732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:36.710595    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:36.798926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:36.798898    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:25:37.140690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.140473    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs" event={"ID":"7cf961da-3329-4396-96f5-817279fd5f03","Type":"ContainerStarted","Data":"23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d"}
Nov 03 13:25:37.141647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.141622    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:37.143111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.143085    5420 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dvwqs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:25:37.143242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.143144    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs" podUID="7cf961da-3329-4396-96f5-817279fd5f03" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:25:37.170613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.170547    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs" podStartSLOduration=1.9598408269999998 podStartE2EDuration="5.170528706s" totalImagesPullingTime="3.210687879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:37.16609722 +0000 UTC m=+978.243862249" watchObservedRunningTime="2026-11-03 13:25:37.170528706 +0000 UTC m=+978.248293731"
Nov 03 13:25:37.205755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.205719    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:37.319776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.319743    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:37.332637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.332613    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:37.348718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.348692    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:37.381130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.381097    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"]
Nov 03 13:25:37.386238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.386189    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021a5a5c-czqpg"]
Nov 03 13:25:37.487257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.487225    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:37.629949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.627432    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"]
Nov 03 13:25:37.629949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.627711    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj" podUID="6528d4c3-5e92-4649-8f21-d4ccab65d2a5" containerName="istio-proxy" containerID="cri-o://bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478" gracePeriod=30
Nov 03 13:25:37.643431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.640854    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:37.803342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:37.803109    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:38.150278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.150178    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:25:38.160601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.160564    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:25:38.287002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.286972    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:38.442261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.441690    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:38.657880 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.657857    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:38.724970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.724942    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:38.764470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764440    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-certs\") pod \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") "
Nov 03 13:25:38.764647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764488    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-podinfo\") pod \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") "
Nov 03 13:25:38.764647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764532    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istiod-ca-cert\") pod \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") "
Nov 03 13:25:38.764647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764571    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-data\") pod \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") "
Nov 03 13:25:38.764824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764677    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-credential-socket\") pod \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") "
Nov 03 13:25:38.764824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764717    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-token\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-token\") pod \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") "
Nov 03 13:25:38.764824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764772    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-envoy\") pod \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") "
Nov 03 13:25:38.764986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764920    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-kube-api-access-cg6bm\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-kube-api-access-cg6bm\") pod \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") "
Nov 03 13:25:38.764986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764766    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-certs" pod "6528d4c3-5e92-4649-8f21-d4ccab65d2a5" (UID: "6528d4c3-5e92-4649-8f21-d4ccab65d2a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:38.765097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764961    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-credential-socket" pod "6528d4c3-5e92-4649-8f21-d4ccab65d2a5" (UID: "6528d4c3-5e92-4649-8f21-d4ccab65d2a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:38.765097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764972    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-data" pod "6528d4c3-5e92-4649-8f21-d4ccab65d2a5" (UID: "6528d4c3-5e92-4649-8f21-d4ccab65d2a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:38.765097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764994    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istiod-ca-cert" pod "6528d4c3-5e92-4649-8f21-d4ccab65d2a5" (UID: "6528d4c3-5e92-4649-8f21-d4ccab65d2a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:38.765097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.764968    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-socket\") pod \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\" (UID: \"6528d4c3-5e92-4649-8f21-d4ccab65d2a5\") "
Nov 03 13:25:38.765336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.765198    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-socket" pod "6528d4c3-5e92-4649-8f21-d4ccab65d2a5" (UID: "6528d4c3-5e92-4649-8f21-d4ccab65d2a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:38.765953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.765926    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:38.766076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.765958    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:38.766076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.765973    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:38.766076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.765989    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:38.766076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.766005    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:38.766854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.766833    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-podinfo" pod "6528d4c3-5e92-4649-8f21-d4ccab65d2a5" (UID: "6528d4c3-5e92-4649-8f21-d4ccab65d2a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:25:38.766966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.766945    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-token" pod "6528d4c3-5e92-4649-8f21-d4ccab65d2a5" (UID: "6528d4c3-5e92-4649-8f21-d4ccab65d2a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:38.767019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.766997    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-kube-api-access-cg6bm" pod "6528d4c3-5e92-4649-8f21-d4ccab65d2a5" (UID: "6528d4c3-5e92-4649-8f21-d4ccab65d2a5"). InnerVolumeSpecName "kube-api-access-cg6bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:38.767059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.767023    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-envoy" pod "6528d4c3-5e92-4649-8f21-d4ccab65d2a5" (UID: "6528d4c3-5e92-4649-8f21-d4ccab65d2a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:25:38.867090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.867063    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:38.867090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.867090    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:38.867505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.867103    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:38.867505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:38.867116    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6bm\" (UniqueName: \"kubernetes.io/projected/6528d4c3-5e92-4649-8f21-d4ccab65d2a5-kube-api-access-cg6bm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:39.087087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.087007    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316b29df-f26e-47e4-b90f-391d5fb7b7fc" path="/var/lib/kubelet/pods/316b29df-f26e-47e4-b90f-391d5fb7b7fc/volumes"
Nov 03 13:25:39.103674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.103648    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:39.160860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.160825    5420 generic.go:356] "Generic (PLEG): container finished" podID="6528d4c3-5e92-4649-8f21-d4ccab65d2a5" containerID="bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478" exitCode=0
Nov 03 13:25:39.161038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.160887    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"
Nov 03 13:25:39.161038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.160914    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj" event={"ID":"6528d4c3-5e92-4649-8f21-d4ccab65d2a5","Type":"ContainerDied","Data":"bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478"}
Nov 03 13:25:39.161038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.160974    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj" event={"ID":"6528d4c3-5e92-4649-8f21-d4ccab65d2a5","Type":"ContainerDied","Data":"c91f2857bf4afa80b5316645944d5585908954b180f53e78b862dbaed3ad802e"}
Nov 03 13:25:39.161038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.161000    5420 scope.go:122] "RemoveContainer" containerID="bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478"
Nov 03 13:25:39.238049 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.238014    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:39.246519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.239133    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"]
Nov 03 13:25:39.248665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.248589    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-021a5a5c-openshift-default-6549bd658f-6kfpj"]
Nov 03 13:25:39.266576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.266542    5420 scope.go:122] "RemoveContainer" containerID="bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478"
Nov 03 13:25:39.267004 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:25:39.266975    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478\": container with ID starting with bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478 not found: ID does not exist" containerID="bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478"
Nov 03 13:25:39.267169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.267018    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478"} err="failed to get container status \"bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478\": rpc error: code = NotFound desc = could not find container \"bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478\": container with ID starting with bc1df725627481ab97c9184f139c755f4f41c8e9ba9439b2a35be9b413eb1478 not found: ID does not exist"
Nov 03 13:25:39.476649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:39.476610    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:40.026403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:40.026371    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"]
Nov 03 13:25:40.032504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:40.032479    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f585f393-hwn5w"]
Nov 03 13:25:40.107923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:40.107889    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:40.416264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:40.416166    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:40.557869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:40.557832    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:40.635728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:40.635686    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw" podUID="91e3e558-7d4a-420e-bdad-8f95e5c220b8" containerName="payload-processing" containerID="cri-o://3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f" gracePeriod=25
Nov 03 13:25:40.697903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:40.697835    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:40.843831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:40.843799    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:40.926024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:40.925963    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:25:41.005645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.005608    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-routing-overlay-volume\") pod \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") "
Nov 03 13:25:41.005845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.005655    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-plugins-config-volume\") pod \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") "
Nov 03 13:25:41.005845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.005786    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-kube-api-access-dfkbm\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-kube-api-access-dfkbm\") pod \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") "
Nov 03 13:25:41.005983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.005961    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-routing-overlay-volume" pod "91e3e558-7d4a-420e-bdad-8f95e5c220b8" (UID: "91e3e558-7d4a-420e-bdad-8f95e5c220b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:41.006048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.005968    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-provider-credentials\") pod \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\" (UID: \"91e3e558-7d4a-420e-bdad-8f95e5c220b8\") "
Nov 03 13:25:41.006107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.006053    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-plugins-config-volume" pod "91e3e558-7d4a-420e-bdad-8f95e5c220b8" (UID: "91e3e558-7d4a-420e-bdad-8f95e5c220b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:41.006563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.006546    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:41.006629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.006564    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e3e558-7d4a-420e-bdad-8f95e5c220b8-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:41.008046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.008013    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-provider-credentials" pod "91e3e558-7d4a-420e-bdad-8f95e5c220b8" (UID: "91e3e558-7d4a-420e-bdad-8f95e5c220b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:41.008142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.008117    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-kube-api-access-dfkbm" pod "91e3e558-7d4a-420e-bdad-8f95e5c220b8" (UID: "91e3e558-7d4a-420e-bdad-8f95e5c220b8"). InnerVolumeSpecName "kube-api-access-dfkbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:41.087955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.087924    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6528d4c3-5e92-4649-8f21-d4ccab65d2a5" path="/var/lib/kubelet/pods/6528d4c3-5e92-4649-8f21-d4ccab65d2a5/volumes"
Nov 03 13:25:41.088488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.088472    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd06a2af-8c53-4106-87ec-f18e2e034c5f" path="/var/lib/kubelet/pods/fd06a2af-8c53-4106-87ec-f18e2e034c5f/volumes"
Nov 03 13:25:41.108290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.108259    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfkbm\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-kube-api-access-dfkbm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:41.108290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.108287    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91e3e558-7d4a-420e-bdad-8f95e5c220b8-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:41.182434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.182343    5420 generic.go:356] "Generic (PLEG): container finished" podID="91e3e558-7d4a-420e-bdad-8f95e5c220b8" containerID="3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f" exitCode=0
Nov 03 13:25:41.182434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.182392    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw" event={"ID":"91e3e558-7d4a-420e-bdad-8f95e5c220b8","Type":"ContainerDied","Data":"3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f"}
Nov 03 13:25:41.182434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.182401    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"
Nov 03 13:25:41.182434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.182437    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw" event={"ID":"91e3e558-7d4a-420e-bdad-8f95e5c220b8","Type":"ContainerDied","Data":"a57c0c0d6ec3dad3c5d6cf941076dbb3e8fb6109e25c331d591b6bb7ddfaaec3"}
Nov 03 13:25:41.182717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.182458    5420 scope.go:122] "RemoveContainer" containerID="3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f"
Nov 03 13:25:41.194009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.193990    5420 scope.go:122] "RemoveContainer" containerID="3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f"
Nov 03 13:25:41.194318 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:25:41.194284    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f\": container with ID starting with 3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f not found: ID does not exist" containerID="3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f"
Nov 03 13:25:41.194406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.194327    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f"} err="failed to get container status \"3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f\": rpc error: code = NotFound desc = could not find container \"3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f\": container with ID starting with 3383cca90f37aa3ad480a9a5d162db1bb2d0a4feb1ada08d00285612b479ae4f not found: ID does not exist"
Nov 03 13:25:41.207096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.207039    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"]
Nov 03 13:25:41.214482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.214451    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-69642v7vw"]
Nov 03 13:25:41.431046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.431004    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:41.875237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:41.869816    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:42.022619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:42.022584    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:25:42.029471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:42.029436    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"]
Nov 03 13:25:42.039730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:42.038939    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0b0fc4-kwzcj"]
Nov 03 13:25:42.246618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:42.245888    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:42.464536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:42.464504    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:43.021388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:43.021357    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:43.059103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:43.059065    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"]
Nov 03 13:25:43.071887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:43.071841    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cead985e-2lq4x"]
Nov 03 13:25:43.087308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:43.087273    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9441c7-9bab-4cd2-aacc-0223c8d1e14f" path="/var/lib/kubelet/pods/7b9441c7-9bab-4cd2-aacc-0223c8d1e14f/volumes"
Nov 03 13:25:43.087897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:43.087881    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8243f0e7-f7b4-4bd4-9158-317a95cd8d72" path="/var/lib/kubelet/pods/8243f0e7-f7b4-4bd4-9158-317a95cd8d72/volumes"
Nov 03 13:25:43.088389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:43.088374    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e3e558-7d4a-420e-bdad-8f95e5c220b8" path="/var/lib/kubelet/pods/91e3e558-7d4a-420e-bdad-8f95e5c220b8/volumes"
Nov 03 13:25:43.220003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:43.219970    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:43.401907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:43.401809    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:43.637392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:43.637357    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:44.020643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:44.020606    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:44.207907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:44.207871    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:44.647084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:44.647050    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:45.424723 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:45.424695    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:46.059321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.059280    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:46.206695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.206662    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:46.289488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.289455    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"]
Nov 03 13:25:46.290111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.290098    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6528d4c3-5e92-4649-8f21-d4ccab65d2a5" containerName="istio-proxy"
Nov 03 13:25:46.290158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.290112    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6528d4c3-5e92-4649-8f21-d4ccab65d2a5" containerName="istio-proxy"
Nov 03 13:25:46.290158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.290136    5420 cpu_manager.go:414] "containerMap: removing container" podUID="91e3e558-7d4a-420e-bdad-8f95e5c220b8" containerName="payload-processing"
Nov 03 13:25:46.290158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.290141    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e3e558-7d4a-420e-bdad-8f95e5c220b8" containerName="payload-processing"
Nov 03 13:25:46.290288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.290280    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6528d4c3-5e92-4649-8f21-d4ccab65d2a5" containerName="istio-proxy"
Nov 03 13:25:46.290323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.290294    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e3e558-7d4a-420e-bdad-8f95e5c220b8" containerName="payload-processing"
Nov 03 13:25:46.297697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.297672    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.302538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.302511    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"]
Nov 03 13:25:46.303325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.303304    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:25:46.303493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.303312    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:25:46.303493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.303327    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:25:46.303682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.303333    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:25:46.407763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.407676    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd278693-z2vrq\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.407763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.407722    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd278693-z2vrq\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.407996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.407878    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6w8s\" (UniqueName: \"kubernetes.io/projected/cf206e3f-d781-4e03-ba7a-3c1b481c8230-kube-api-access-c6w8s\") pod \"maas-api-revoke-sub-dd278693-z2vrq\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.509071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.509032    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6w8s\" (UniqueName: \"kubernetes.io/projected/cf206e3f-d781-4e03-ba7a-3c1b481c8230-kube-api-access-c6w8s\") pod \"maas-api-revoke-sub-dd278693-z2vrq\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.509359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.509329    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd278693-z2vrq\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.509432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.509377    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd278693-z2vrq\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.510259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.510232    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd278693-z2vrq\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.522985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.522951    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd278693-z2vrq\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.526054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.526024    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6w8s\" (UniqueName: \"kubernetes.io/projected/cf206e3f-d781-4e03-ba7a-3c1b481c8230-kube-api-access-c6w8s\") pod \"maas-api-revoke-sub-dd278693-z2vrq\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.617851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.617812    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:46.768749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.768714    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:46.923715 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:25:46.923678    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf206e3f_d781_4e03_ba7a_3c1b481c8230.slice/crio-54a562c25567f695067cc4e76e5bd9b44d720e52ed767575d2b4ba5323e0183b WatchSource:0}: Error finding container 54a562c25567f695067cc4e76e5bd9b44d720e52ed767575d2b4ba5323e0183b: Status 404 returned error can't find the container with id 54a562c25567f695067cc4e76e5bd9b44d720e52ed767575d2b4ba5323e0183b
Nov 03 13:25:46.924670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:46.924649    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"]
Nov 03 13:25:47.157361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:47.157310    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:47.244625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:47.244593    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq" event={"ID":"cf206e3f-d781-4e03-ba7a-3c1b481c8230","Type":"ContainerStarted","Data":"54a562c25567f695067cc4e76e5bd9b44d720e52ed767575d2b4ba5323e0183b"}
Nov 03 13:25:47.306842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:47.306799    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:47.875838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:47.875802    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:48.216228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:48.216182    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:48.257355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:48.257322    5420 generic.go:356] "Generic (PLEG): container finished" podID="cf206e3f-d781-4e03-ba7a-3c1b481c8230" containerID="b40886127078d31ab74a5ade7f99b78c675e44f2be8a31777f27a0a4a83a3d1b" exitCode=0
Nov 03 13:25:48.257529 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:48.257384    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq" event={"ID":"cf206e3f-d781-4e03-ba7a-3c1b481c8230","Type":"ContainerDied","Data":"b40886127078d31ab74a5ade7f99b78c675e44f2be8a31777f27a0a4a83a3d1b"}
Nov 03 13:25:48.493062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:48.492970    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:48.643864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:48.643757    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:49.117104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.117013    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:49.434590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.434568    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:49.555756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.555720    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-openshift-ca\") pod \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") "
Nov 03 13:25:49.555953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.555886    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-xks-ca\") pod \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") "
Nov 03 13:25:49.556033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.555987    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf206e3f-d781-4e03-ba7a-3c1b481c8230-kube-api-access-c6w8s\" (UniqueName: \"kubernetes.io/projected/cf206e3f-d781-4e03-ba7a-3c1b481c8230-kube-api-access-c6w8s\") pod \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\" (UID: \"cf206e3f-d781-4e03-ba7a-3c1b481c8230\") "
Nov 03 13:25:49.556142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.556113    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-openshift-ca" pod "cf206e3f-d781-4e03-ba7a-3c1b481c8230" (UID: "cf206e3f-d781-4e03-ba7a-3c1b481c8230"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:25:49.556571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.556549    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:49.557984 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.557953    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf206e3f-d781-4e03-ba7a-3c1b481c8230-kube-api-access-c6w8s" pod "cf206e3f-d781-4e03-ba7a-3c1b481c8230" (UID: "cf206e3f-d781-4e03-ba7a-3c1b481c8230"). InnerVolumeSpecName "kube-api-access-c6w8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:25:49.558078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.558005    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-xks-ca" pod "cf206e3f-d781-4e03-ba7a-3c1b481c8230" (UID: "cf206e3f-d781-4e03-ba7a-3c1b481c8230"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:25:49.658273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.658165    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf206e3f-d781-4e03-ba7a-3c1b481c8230-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:49.658273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.658195    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6w8s\" (UniqueName: \"kubernetes.io/projected/cf206e3f-d781-4e03-ba7a-3c1b481c8230-kube-api-access-c6w8s\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:25:49.879548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.879513    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:25:49.913620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:49.913519    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:50.279663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:50.279633    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"
Nov 03 13:25:50.279663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:50.279643    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq" event={"ID":"cf206e3f-d781-4e03-ba7a-3c1b481c8230","Type":"ContainerDied","Data":"54a562c25567f695067cc4e76e5bd9b44d720e52ed767575d2b4ba5323e0183b"}
Nov 03 13:25:50.279883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:50.279697    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54a562c25567f695067cc4e76e5bd9b44d720e52ed767575d2b4ba5323e0183b"
Nov 03 13:25:50.811609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:50.811573    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:51.741048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:51.738808    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:52.417064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:52.417036    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:52.579984 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:52.579947    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:53.513763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:53.513726    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:54.027770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:54.027740    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"]
Nov 03 13:25:54.035788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:54.035756    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4074262-gstsl"]
Nov 03 13:25:54.041016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:54.040994    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"]
Nov 03 13:25:54.046778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:54.046748    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec84dd02-8mntd"]
Nov 03 13:25:54.289792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:54.289688    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:54.682528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:54.682441    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:54.740038 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:25:54.740006    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:25:54.839789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:54.839753    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:55.015668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:55.015636    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:55.088544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:55.088507    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bb4563-2d22-452e-ae90-68c8c477ce7a" path="/var/lib/kubelet/pods/59bb4563-2d22-452e-ae90-68c8c477ce7a/volumes"
Nov 03 13:25:55.089059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:55.089044    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44ac396-14d6-42ff-8388-b440acd7aa46" path="/var/lib/kubelet/pods/c44ac396-14d6-42ff-8388-b440acd7aa46/volumes"
Nov 03 13:25:55.178450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:55.178410    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:55.800848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:55.800815    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:55.954525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:55.954496    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:56.896557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:56.896472    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:57.058005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:57.057973    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:57.911360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:57.911324    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:58.035315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:58.035268    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"]
Nov 03 13:25:58.040298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:58.040269    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b9a106a-8rfqf"]
Nov 03 13:25:58.366340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:58.366304    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:58.603396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:58.603356    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:59.085473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:59.085440    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4365d6b0-ca51-4863-9ad8-d855850fcead" path="/var/lib/kubelet/pods/4365d6b0-ca51-4863-9ad8-d855850fcead/volumes"
Nov 03 13:25:59.297703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:25:59.297673    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:00.076662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:00.076619    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:00.824760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:00.824710    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:01.498064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:01.498025    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:02.292110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:02.292066    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:02.962639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:02.962592    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:03.766370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:03.766333    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:04.318064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:04.318025    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:04.916596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:04.916551    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:05.504424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:05.504387    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:06.044244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:06.040591    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"]
Nov 03 13:26:06.049341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:06.049307    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0b9ad-cjq9w"]
Nov 03 13:26:06.211484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:06.211449    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:06.784187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:06.784156    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:07.028610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.028577    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"]
Nov 03 13:26:07.041561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.041474    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8ead43-jf5d8"]
Nov 03 13:26:07.085981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.085953    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0589a5ea-bf21-48fd-8534-8e6866a2f88a" path="/var/lib/kubelet/pods/0589a5ea-bf21-48fd-8534-8e6866a2f88a/volumes"
Nov 03 13:26:07.086451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.086437    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff68e187-075b-46fb-8977-025f0f0123b7" path="/var/lib/kubelet/pods/ff68e187-075b-46fb-8977-025f0f0123b7/volumes"
Nov 03 13:26:07.145945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.145888    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:07.421205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.421080    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:07.940243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.940199    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"]
Nov 03 13:26:07.940873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.940859    5420 cpu_manager.go:414] "containerMap: removing container" podUID="cf206e3f-d781-4e03-ba7a-3c1b481c8230" containerName="revoke-keys"
Nov 03 13:26:07.940925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.940876    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf206e3f-d781-4e03-ba7a-3c1b481c8230" containerName="revoke-keys"
Nov 03 13:26:07.941010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.941001    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf206e3f-d781-4e03-ba7a-3c1b481c8230" containerName="revoke-keys"
Nov 03 13:26:07.946191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.946168    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:07.952393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.952370    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:26:07.952573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.952415    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:26:07.954041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.954014    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"]
Nov 03 13:26:07.963312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.963005    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:26:07.963312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:07.963086    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:26:08.037477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.037422    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:08.126934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.126899    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.127358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.127060    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72299\" (UniqueName: \"kubernetes.io/projected/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-kube-api-access-72299\") pod \"maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.127358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.127097    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.228719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.228686    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72299\" (UniqueName: \"kubernetes.io/projected/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-kube-api-access-72299\") pod \"maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.228719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.228725    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.229000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.228895    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.229603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.229581    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.231202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.231180    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.237812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.237792    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72299\" (UniqueName: \"kubernetes.io/projected/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-kube-api-access-72299\") pod \"maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.274936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.274909    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:08.565154 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:26:08.565113    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfa0a4e_5872_47c2_bdec_edd9f85fbdca.slice/crio-760349801a05d6f452120c72abb5738ae64f5db54f4872fa2416816ace5cec14 WatchSource:0}: Error finding container 760349801a05d6f452120c72abb5738ae64f5db54f4872fa2416816ace5cec14: Status 404 returned error can't find the container with id 760349801a05d6f452120c72abb5738ae64f5db54f4872fa2416816ace5cec14
Nov 03 13:26:08.566910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.566891    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"]
Nov 03 13:26:08.920457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:08.920417    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:09.381361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:09.381326    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:26:09.506886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:09.506855    5420 generic.go:356] "Generic (PLEG): container finished" podID="0cfa0a4e-5872-47c2-bdec-edd9f85fbdca" containerID="6c816b252f0f49141542af8de21eb0ab6b71eafdb3ed5bfce52122adb3d1d8e1" exitCode=0
Nov 03 13:26:09.507051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:09.506937    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz" event={"ID":"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca","Type":"ContainerDied","Data":"6c816b252f0f49141542af8de21eb0ab6b71eafdb3ed5bfce52122adb3d1d8e1"}
Nov 03 13:26:09.507051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:09.506987    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz" event={"ID":"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca","Type":"ContainerStarted","Data":"760349801a05d6f452120c72abb5738ae64f5db54f4872fa2416816ace5cec14"}
Nov 03 13:26:09.574874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:09.574833    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:10.176592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:10.176563    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:10.635263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:10.632676    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:10.967235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:10.966421    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:11.048811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.048779    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-xks-ca\") pod \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") "
Nov 03 13:26:11.055319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.055281    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-kube-api-access-72299\" (UniqueName: \"kubernetes.io/projected/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-kube-api-access-72299\") pod \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") "
Nov 03 13:26:11.082769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.082620    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-openshift-ca\") pod \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\" (UID: \"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca\") "
Nov 03 13:26:11.083891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.083698    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-openshift-ca" pod "0cfa0a4e-5872-47c2-bdec-edd9f85fbdca" (UID: "0cfa0a4e-5872-47c2-bdec-edd9f85fbdca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:11.090271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.090187    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:11.159248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.159156    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-xks-ca" pod "0cfa0a4e-5872-47c2-bdec-edd9f85fbdca" (UID: "0cfa0a4e-5872-47c2-bdec-edd9f85fbdca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:11.167029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.166988    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-kube-api-access-72299" pod "0cfa0a4e-5872-47c2-bdec-edd9f85fbdca" (UID: "0cfa0a4e-5872-47c2-bdec-edd9f85fbdca"). InnerVolumeSpecName "kube-api-access-72299". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:11.199084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.199046    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:11.199084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.199086    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72299\" (UniqueName: \"kubernetes.io/projected/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca-kube-api-access-72299\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:11.518882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.518846    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:11.540472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.540445    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"
Nov 03 13:26:11.540472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.540450    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz" event={"ID":"0cfa0a4e-5872-47c2-bdec-edd9f85fbdca","Type":"ContainerDied","Data":"760349801a05d6f452120c72abb5738ae64f5db54f4872fa2416816ace5cec14"}
Nov 03 13:26:11.540754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:11.540515    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760349801a05d6f452120c72abb5738ae64f5db54f4872fa2416816ace5cec14"
Nov 03 13:26:12.228048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:12.228013    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:12.880124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:12.880086    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:13.520090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:13.520059    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:14.201393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:14.201354    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:15.000833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:15.000765    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:15.725220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:15.725174    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:16.498986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:16.498948    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:17.125063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:17.125025    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:18.537968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:18.537939    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:18.765876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:18.765847    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:19.132630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:19.132593    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:19.383358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:19.383260    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:19.469497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:19.469457    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"]
Nov 03 13:26:19.474299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:19.474271    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2dbf739c-v79vz"]
Nov 03 13:26:19.551592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:19.551560    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:19.724065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:19.724030    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:20.151235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.151133    5420 scope.go:122] "RemoveContainer" containerID="daf4d0cbf08dd9d1d88cd119b82f5210ed8fb19612d9eac10b0bd076585cacd9"
Nov 03 13:26:20.162503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.162482    5420 scope.go:122] "RemoveContainer" containerID="5f68363568338d31221b112fc247846b7bfdf93f995dfa6592a88a756a1aa94e"
Nov 03 13:26:20.172832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.172811    5420 scope.go:122] "RemoveContainer" containerID="c2e1249ff41a131e04974b1211eea5085252d0b531705a2556411d35cba57f16"
Nov 03 13:26:20.184875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.184851    5420 scope.go:122] "RemoveContainer" containerID="af08b388fe7ab32b80a6270b40a8493306d780f50053684a7fb8ed3e1b63d6fc"
Nov 03 13:26:20.194712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.194691    5420 scope.go:122] "RemoveContainer" containerID="d10d7cc1ff97d04cea21c8e93efa1f4171ca15c2f9745ab5eaa8aa324b57da3f"
Nov 03 13:26:20.205544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.205527    5420 scope.go:122] "RemoveContainer" containerID="e3f7ee5f1c315dd05971796638a798a300526e541174ce4868d0daa62012c50c"
Nov 03 13:26:20.215700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.215683    5420 scope.go:122] "RemoveContainer" containerID="d52e218e3f42ab743a80b05dd84a15381ca60f183b8828ebc4bb737be8b3082a"
Nov 03 13:26:20.225931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.225910    5420 scope.go:122] "RemoveContainer" containerID="18fa6b782137c494936efad535baf923e1d2c5a1de8669b1a9093dc6b6b7f556"
Nov 03 13:26:20.306824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.306801    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:20.452638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.452614    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:20.663772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.663739    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"]
Nov 03 13:26:20.664540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.664522    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0cfa0a4e-5872-47c2-bdec-edd9f85fbdca" containerName="revoke-keys"
Nov 03 13:26:20.664540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.664541    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cfa0a4e-5872-47c2-bdec-edd9f85fbdca" containerName="revoke-keys"
Nov 03 13:26:20.664692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.664686    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cfa0a4e-5872-47c2-bdec-edd9f85fbdca" containerName="revoke-keys"
Nov 03 13:26:20.670350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.670329    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.678028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.678003    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e04a4e0c-openshift-default-dockercfg-k9tzd\""
Nov 03 13:26:20.685731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.685708    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"]
Nov 03 13:26:20.751910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.751832    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-credential-socket\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.751910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.751864    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-data\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.751910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.751889    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2a373-b468-4904-9dff-770419fc4e2a-istiod-ca-cert\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.752157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.751972    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-socket\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.752157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.752092    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-token\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.752157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.752144    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcl4v\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-kube-api-access-zcl4v\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.752348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.752253    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-envoy\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.752348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.752311    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-podinfo\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.752452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.752401    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-certs\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.755852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.755828    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:20.853607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.853575    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-credential-socket\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.853607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.853616    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-data\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.853870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.853651    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2a373-b468-4904-9dff-770419fc4e2a-istiod-ca-cert\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.853870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.853684    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-socket\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.853870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.853733    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-token\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.853870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.853800    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcl4v\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-kube-api-access-zcl4v\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.853870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.853837    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-envoy\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.854181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.853890    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-podinfo\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.854181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.853974    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-certs\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.854181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.854024    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-credential-socket\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.854427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.854260    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-socket\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.854571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.854541    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-data\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.854670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.854650    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2a373-b468-4904-9dff-770419fc4e2a-istiod-ca-cert\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.855399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.855371    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-certs\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.857641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.857595    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-envoy\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.889232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.876294    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-podinfo\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.889232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.883910    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-token\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.890118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.890086    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcl4v\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-kube-api-access-zcl4v\") pod \"e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") " pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:20.943247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.935815    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:20.989164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:20.989132    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:21.032232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.030988    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"]
Nov 03 13:26:21.039445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.039411    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfff674b-z9clp"]
Nov 03 13:26:21.103059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.103020    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056d541c-e3b4-44a5-9d8d-74360e94a576" path="/var/lib/kubelet/pods/056d541c-e3b4-44a5-9d8d-74360e94a576/volumes"
Nov 03 13:26:21.104163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.104137    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cfa0a4e-5872-47c2-bdec-edd9f85fbdca" path="/var/lib/kubelet/pods/0cfa0a4e-5872-47c2-bdec-edd9f85fbdca/volumes"
Nov 03 13:26:21.206171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.206139    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:21.302731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.302505    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"]
Nov 03 13:26:21.305491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.305454    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:21.305612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.305522    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:21.305612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.305577    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:21.663717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.663626    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4" event={"ID":"ccc2a373-b468-4904-9dff-770419fc4e2a","Type":"ContainerStarted","Data":"0c23c8c70506a936284bacc82dc39964ee3e413d80a35857c0877888cd85b9d2"}
Nov 03 13:26:21.663717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.663683    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4" event={"ID":"ccc2a373-b468-4904-9dff-770419fc4e2a","Type":"ContainerStarted","Data":"7ff639eb47828f80ce9f9866cef4e1c6ecd5617e02c2633644d5f109908d46cb"}
Nov 03 13:26:21.692082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.692027    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4" podStartSLOduration=1.6920131010000001 podStartE2EDuration="1.692013101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:21.688354587 +0000 UTC m=+1022.766119625" watchObservedRunningTime="2026-11-03 13:26:21.692013101 +0000 UTC m=+1022.769778126"
Nov 03 13:26:21.983224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.983166    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:21.989597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.989575    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:21.996363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:21.996338    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"]
Nov 03 13:26:22.001479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.001457    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.010300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.008414    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:26:22.010300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.008569    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:26:22.010300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.008638    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:26:22.010300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.008714    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:26:22.018526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.018496    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"]
Nov 03 13:26:22.040226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.040174    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"]
Nov 03 13:26:22.049571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.049532    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eee2e40-k6sz8"]
Nov 03 13:26:22.082486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.082455    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-669d50fc-cnlfb\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.082674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.082515    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rzx\" (UniqueName: \"kubernetes.io/projected/beb086d0-ad43-449a-98f7-42b6fbc19195-kube-api-access-f5rzx\") pod \"maas-api-revoke-sub-669d50fc-cnlfb\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.082749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.082696    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-669d50fc-cnlfb\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.183760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.183721    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-669d50fc-cnlfb\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.183968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.183821    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rzx\" (UniqueName: \"kubernetes.io/projected/beb086d0-ad43-449a-98f7-42b6fbc19195-kube-api-access-f5rzx\") pod \"maas-api-revoke-sub-669d50fc-cnlfb\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.184043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.184006    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-669d50fc-cnlfb\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.184561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.184535    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-669d50fc-cnlfb\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.186517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.186492    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-669d50fc-cnlfb\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.193706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.193682    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rzx\" (UniqueName: \"kubernetes.io/projected/beb086d0-ad43-449a-98f7-42b6fbc19195-kube-api-access-f5rzx\") pod \"maas-api-revoke-sub-669d50fc-cnlfb\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.323603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.323524    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:22.610253 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:26:22.610221    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb086d0_ad43_449a_98f7_42b6fbc19195.slice/crio-ce7b093991860fd7f2f4f75d24bce46f3b65647c7b003ba4ad489aab69435d64 WatchSource:0}: Error finding container ce7b093991860fd7f2f4f75d24bce46f3b65647c7b003ba4ad489aab69435d64: Status 404 returned error can't find the container with id ce7b093991860fd7f2f4f75d24bce46f3b65647c7b003ba4ad489aab69435d64
Nov 03 13:26:22.611968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.611942    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"]
Nov 03 13:26:22.673997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.673959    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb" event={"ID":"beb086d0-ad43-449a-98f7-42b6fbc19195","Type":"ContainerStarted","Data":"ce7b093991860fd7f2f4f75d24bce46f3b65647c7b003ba4ad489aab69435d64"}
Nov 03 13:26:22.820498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.820460    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:22.995481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:22.995450    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:23.086437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:23.086405    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef92699-edad-4df1-98c0-ef51e8c07891" path="/var/lib/kubelet/pods/eef92699-edad-4df1-98c0-ef51e8c07891/volumes"
Nov 03 13:26:23.318676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:23.318624    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:23.521204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:23.521172    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:23.686959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:23.686921    5420 generic.go:356] "Generic (PLEG): container finished" podID="beb086d0-ad43-449a-98f7-42b6fbc19195" containerID="9ec4c50f99f9b7e5c4ce00f90741a2587411c92780fe732987aad22aed4656bf" exitCode=0
Nov 03 13:26:23.687558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:23.687059    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb" event={"ID":"beb086d0-ad43-449a-98f7-42b6fbc19195","Type":"ContainerDied","Data":"9ec4c50f99f9b7e5c4ce00f90741a2587411c92780fe732987aad22aed4656bf"}
Nov 03 13:26:23.687631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:23.687564    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:23.687631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:23.687616    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:23.688611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:23.688594    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:26:24.625499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:24.625468    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:24.744809 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:26:24.744760    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:26:24.789022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:24.788989    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:25.016809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.016783    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:25.195760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.195725    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-openshift-ca\") pod \"beb086d0-ad43-449a-98f7-42b6fbc19195\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") "
Nov 03 13:26:25.195959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.195884    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-xks-ca\") pod \"beb086d0-ad43-449a-98f7-42b6fbc19195\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") "
Nov 03 13:26:25.196015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.195992    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb086d0-ad43-449a-98f7-42b6fbc19195-kube-api-access-f5rzx\" (UniqueName: \"kubernetes.io/projected/beb086d0-ad43-449a-98f7-42b6fbc19195-kube-api-access-f5rzx\") pod \"beb086d0-ad43-449a-98f7-42b6fbc19195\" (UID: \"beb086d0-ad43-449a-98f7-42b6fbc19195\") "
Nov 03 13:26:25.196162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.196133    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-openshift-ca" pod "beb086d0-ad43-449a-98f7-42b6fbc19195" (UID: "beb086d0-ad43-449a-98f7-42b6fbc19195"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:25.196697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.196661    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:25.197975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.197951    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-xks-ca" pod "beb086d0-ad43-449a-98f7-42b6fbc19195" (UID: "beb086d0-ad43-449a-98f7-42b6fbc19195"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:25.198250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.198230    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb086d0-ad43-449a-98f7-42b6fbc19195-kube-api-access-f5rzx" pod "beb086d0-ad43-449a-98f7-42b6fbc19195" (UID: "beb086d0-ad43-449a-98f7-42b6fbc19195"). InnerVolumeSpecName "kube-api-access-f5rzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:25.298017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.297920    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5rzx\" (UniqueName: \"kubernetes.io/projected/beb086d0-ad43-449a-98f7-42b6fbc19195-kube-api-access-f5rzx\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:25.298017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.297966    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb086d0-ad43-449a-98f7-42b6fbc19195-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:25.705415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.705387    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"
Nov 03 13:26:25.705557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.705412    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb" event={"ID":"beb086d0-ad43-449a-98f7-42b6fbc19195","Type":"ContainerDied","Data":"ce7b093991860fd7f2f4f75d24bce46f3b65647c7b003ba4ad489aab69435d64"}
Nov 03 13:26:25.705557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.705481    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7b093991860fd7f2f4f75d24bce46f3b65647c7b003ba4ad489aab69435d64"
Nov 03 13:26:25.732442 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:25.732394    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:26.034618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:26.034586    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"]
Nov 03 13:26:26.039575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:26.039551    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a39b1980-2h2xm"]
Nov 03 13:26:26.052727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:26.052706    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:26.471852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:26.471821    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:27.086104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:27.086072    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c204ac-92cf-4322-8dff-c8c1ae3c5100" path="/var/lib/kubelet/pods/45c204ac-92cf-4322-8dff-c8c1ae3c5100/volumes"
Nov 03 13:26:27.203998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:27.203957    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:27.840885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:27.840853    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:28.116971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.116898    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"]
Nov 03 13:26:28.117566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.117551    5420 cpu_manager.go:414] "containerMap: removing container" podUID="beb086d0-ad43-449a-98f7-42b6fbc19195" containerName="revoke-keys"
Nov 03 13:26:28.117618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.117566    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb086d0-ad43-449a-98f7-42b6fbc19195" containerName="revoke-keys"
Nov 03 13:26:28.117707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.117696    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb086d0-ad43-449a-98f7-42b6fbc19195" containerName="revoke-keys"
Nov 03 13:26:28.122581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.122565    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.131447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.131421    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9add72-openshift-default-dockercfg-c6nrt\""
Nov 03 13:26:28.134676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.134652    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"]
Nov 03 13:26:28.238903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.238874    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.239071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.238935    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.239071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.238996    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.239247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.239193    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.239509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.239489    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.239620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.239607    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.239688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.239634    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.239688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.239656    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.239774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.239716    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mznf\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-kube-api-access-2mznf\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.340832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.340798    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.340993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.340874    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.340993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.340894    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.340993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.340918    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.340993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.340964    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mznf\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-kube-api-access-2mznf\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.341198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.341027    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.341198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.341072    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.341198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.341114    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.341198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.341143    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.341415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.341297    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.341415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.341357    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.341519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.341482    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.341573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.341538    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.341746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.341726    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.343314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.343287    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.343803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.343781    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.349695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.349676    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mznf\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-kube-api-access-2mznf\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.349890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.349872    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.444700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.444634    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:28.607136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.607099    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:28.741648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.741616    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"]
Nov 03 13:26:28.748130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.746255    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:28.748130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.746392    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:28.748130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:28.746477    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:29.196191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:29.196089    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:29.753753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:29.753709    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" event={"ID":"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9","Type":"ContainerStarted","Data":"22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d"}
Nov 03 13:26:29.753940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:29.753767    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" event={"ID":"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9","Type":"ContainerStarted","Data":"0e86b92de249cfa612a5db8a7ce0b770dd25bd952f42758fde229f6515b71c33"}
Nov 03 13:26:29.775320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:29.775277    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" podStartSLOduration=1.775264862 podStartE2EDuration="1.775264862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:29.774421982 +0000 UTC m=+1030.852187005" watchObservedRunningTime="2026-11-03 13:26:29.775264862 +0000 UTC m=+1030.853029887"
Nov 03 13:26:29.909377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:29.909348    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:30.445404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:30.445373    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:30.447199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:30.447167    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 13:26:30.447341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:30.447245    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 13:26:30.644394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:30.641200    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:31.451631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:31.451605    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 13:26:31.452080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:31.452057    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 13:26:31.452256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:31.451987    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:31.584261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:31.584197    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:31.717584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:31.717555    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:32.445628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:32.445598    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 13:26:32.445796 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:32.445645    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 13:26:32.629254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:32.629201    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:33.050468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:33.050429    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"]
Nov 03 13:26:33.055077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:33.055052    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"]
Nov 03 13:26:33.059295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:33.059270    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-444ae124-7v2gn"]
Nov 03 13:26:33.063164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:33.063146    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-283868ba-nkpmc"]
Nov 03 13:26:33.085890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:33.085860    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02cd03f8-4f08-437b-8f56-1bc54b840aff" path="/var/lib/kubelet/pods/02cd03f8-4f08-437b-8f56-1bc54b840aff/volumes"
Nov 03 13:26:33.086340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:33.086325    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a1d702-1b69-48fb-ae4b-8fd168912f17" path="/var/lib/kubelet/pods/65a1d702-1b69-48fb-ae4b-8fd168912f17/volumes"
Nov 03 13:26:33.445858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:33.445771    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 13:26:33.446119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:33.446083    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 13:26:33.535231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:33.531749    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:34.315116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:34.315077    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:34.445702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:34.445653    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 13:26:34.445917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:34.445720    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 13:26:35.038070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:35.038035    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"]
Nov 03 13:26:35.042640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:35.042606    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b554560c-67gh6"]
Nov 03 13:26:35.088724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:35.088688    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a053c9-9607-466c-8a69-b0c2f64ac4f0" path="/var/lib/kubelet/pods/f5a053c9-9607-466c-8a69-b0c2f64ac4f0/volumes"
Nov 03 13:26:35.281678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:35.281650    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:35.450705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:35.450629    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:35.451154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:35.451133    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:35.451795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:35.451770    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:26:36.026251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.026200    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:36.294261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.294161    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"]
Nov 03 13:26:36.300552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.300528    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.316928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.316901    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"]
Nov 03 13:26:36.395339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.395311    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwjzq\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-kube-api-access-wwjzq\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.395507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.395372    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.395507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.395463    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b513df0e-7f3b-4d73-85c8-e7a7830af016-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.395507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.395500    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.395639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.395522    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.395691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.395667    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.395752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.395736    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.395858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.395839    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.395914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.395889    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497299    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497338    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497409    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497465    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497570    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497749    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497725    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497806    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497847    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwjzq\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-kube-api-access-wwjzq\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497898    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.497962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497929    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.498245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.497989    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b513df0e-7f3b-4d73-85c8-e7a7830af016-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.498327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.498256    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.498772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.498744    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b513df0e-7f3b-4d73-85c8-e7a7830af016-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.500088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.500066    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.500300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.500278    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.505786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.505763    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.506032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.506015    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwjzq\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-kube-api-access-wwjzq\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.625079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.625008    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:36.706497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.706435    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:36.914860 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:26:36.914833    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb513df0e_7f3b_4d73_85c8_e7a7830af016.slice/crio-5f32c2babcc25aab7d6fb069bcabb4cbc3acbc171341d73f8ef447de980f5e8f WatchSource:0}: Error finding container 5f32c2babcc25aab7d6fb069bcabb4cbc3acbc171341d73f8ef447de980f5e8f: Status 404 returned error can't find the container with id 5f32c2babcc25aab7d6fb069bcabb4cbc3acbc171341d73f8ef447de980f5e8f
Nov 03 13:26:36.916070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.916046    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"]
Nov 03 13:26:36.917902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.917866    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:36.917971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.917935    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:36.917971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:36.917964    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:37.298314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:37.298283    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:37.831140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:37.831091    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" event={"ID":"b513df0e-7f3b-4d73-85c8-e7a7830af016","Type":"ContainerStarted","Data":"d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe"}
Nov 03 13:26:37.831718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:37.831174    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" event={"ID":"b513df0e-7f3b-4d73-85c8-e7a7830af016","Type":"ContainerStarted","Data":"5f32c2babcc25aab7d6fb069bcabb4cbc3acbc171341d73f8ef447de980f5e8f"}
Nov 03 13:26:37.861612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:37.861551    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" podStartSLOduration=1.861533002 podStartE2EDuration="1.861533002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:37.858014197 +0000 UTC m=+1038.935779222" watchObservedRunningTime="2026-11-03 13:26:37.861533002 +0000 UTC m=+1038.939298029"
Nov 03 13:26:38.012130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:38.012086    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:38.618386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:38.618352    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:38.629842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:38.629809    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:38.630161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:38.630138    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 13:26:38.630407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:38.630380    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 13:26:39.209329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:39.209296    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:39.643292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:39.642659    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 13:26:39.643292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:39.642721    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 13:26:39.826201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:39.826161    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:40.419491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:40.419462    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:40.632648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:40.632611    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 13:26:40.632913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:40.632886    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 13:26:41.107348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:41.107310    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:41.645736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:41.645698    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 13:26:41.646143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:41.645777    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 13:26:41.802546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:41.802463    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:42.518487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.518453    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"]
Nov 03 13:26:42.543495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.543471    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.551301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.549504    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:26:42.551301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.550127    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:26:42.551301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.550296    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:26:42.551301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.550390    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:26:42.552718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.552685    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"]
Nov 03 13:26:42.616595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.616556    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3aa55a0-b46lj\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.617092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.617065    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7wz\" (UniqueName: \"kubernetes.io/projected/15437773-5d9d-4743-ae1b-5b84f43c1daf-kube-api-access-lb7wz\") pod \"maas-api-revoke-sub-b3aa55a0-b46lj\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.617363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.617332    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3aa55a0-b46lj\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.642234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.628630    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 13:26:42.642234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.628713    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 13:26:42.644862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.644836    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:42.719325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.719289    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3aa55a0-b46lj\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.719807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.719413    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3aa55a0-b46lj\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.719807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.719562    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7wz\" (UniqueName: \"kubernetes.io/projected/15437773-5d9d-4743-ae1b-5b84f43c1daf-kube-api-access-lb7wz\") pod \"maas-api-revoke-sub-b3aa55a0-b46lj\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.728153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.720455    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3aa55a0-b46lj\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.749232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.749187    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3aa55a0-b46lj\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.756533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.756511    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7wz\" (UniqueName: \"kubernetes.io/projected/15437773-5d9d-4743-ae1b-5b84f43c1daf-kube-api-access-lb7wz\") pod \"maas-api-revoke-sub-b3aa55a0-b46lj\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:42.865720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:42.865638    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:43.177393 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:26:43.177365    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15437773_5d9d_4743_ae1b_5b84f43c1daf.slice/crio-98e6c40ad089c6dc902cace06f688d1511bf670bc8017ddaa3e74a4341fd3528 WatchSource:0}: Error finding container 98e6c40ad089c6dc902cace06f688d1511bf670bc8017ddaa3e74a4341fd3528: Status 404 returned error can't find the container with id 98e6c40ad089c6dc902cace06f688d1511bf670bc8017ddaa3e74a4341fd3528
Nov 03 13:26:43.178675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:43.178654    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"]
Nov 03 13:26:43.324106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:43.324070    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:43.626320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:43.626284    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 13:26:43.626495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:43.626345    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 13:26:43.905607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:43.905526    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:43.923751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:43.923713    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj" event={"ID":"15437773-5d9d-4743-ae1b-5b84f43c1daf","Type":"ContainerStarted","Data":"2598fca336410e7fc3e801ee6807c1ef936e8bb317296af49034599d7af65fa4"}
Nov 03 13:26:43.923955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:43.923774    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj" event={"ID":"15437773-5d9d-4743-ae1b-5b84f43c1daf","Type":"ContainerStarted","Data":"98e6c40ad089c6dc902cace06f688d1511bf670bc8017ddaa3e74a4341fd3528"}
Nov 03 13:26:43.949195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:43.949132    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj" podStartSLOduration=1.949114564 podStartE2EDuration="1.949114564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:43.941442332 +0000 UTC m=+1045.019207361" watchObservedRunningTime="2026-11-03 13:26:43.949114564 +0000 UTC m=+1045.026879591"
Nov 03 13:26:44.554282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.551023    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:44.572554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.566285    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"]
Nov 03 13:26:44.572554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.572514    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.576781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.572900    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"]
Nov 03 13:26:44.642349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.642315    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:44.642856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.642832    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:44.643833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.643810    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:26:44.680766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.680740    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.680907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.680770    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wwr\" (UniqueName: \"kubernetes.io/projected/b49c2905-88aa-496f-924c-c94e8818dd7f-kube-api-access-v7wwr\") pod \"maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.680907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.680885    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.782624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.782588    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.782905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.782873    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.783022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.782907    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wwr\" (UniqueName: \"kubernetes.io/projected/b49c2905-88aa-496f-924c-c94e8818dd7f-kube-api-access-v7wwr\") pod \"maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.783512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.783489    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.784962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.784942    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.791782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.791761    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wwr\" (UniqueName: \"kubernetes.io/projected/b49c2905-88aa-496f-924c-c94e8818dd7f-kube-api-access-v7wwr\") pod \"maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.912467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.912387    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:44.937483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.937450    5420 generic.go:356] "Generic (PLEG): container finished" podID="15437773-5d9d-4743-ae1b-5b84f43c1daf" containerID="2598fca336410e7fc3e801ee6807c1ef936e8bb317296af49034599d7af65fa4" exitCode=0
Nov 03 13:26:44.937602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:44.937533    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj" event={"ID":"15437773-5d9d-4743-ae1b-5b84f43c1daf","Type":"ContainerDied","Data":"2598fca336410e7fc3e801ee6807c1ef936e8bb317296af49034599d7af65fa4"}
Nov 03 13:26:45.202935 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:26:45.202902    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49c2905_88aa_496f_924c_c94e8818dd7f.slice/crio-33c53d530338816e9f0dc0d4cb2f21d488548ce479969cb334b57df4722f2113 WatchSource:0}: Error finding container 33c53d530338816e9f0dc0d4cb2f21d488548ce479969cb334b57df4722f2113: Status 404 returned error can't find the container with id 33c53d530338816e9f0dc0d4cb2f21d488548ce479969cb334b57df4722f2113
Nov 03 13:26:45.203907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:45.203884    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"]
Nov 03 13:26:45.422116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:45.422069    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:45.727111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:45.727075    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:45.868443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:45.868364    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:45.948912 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:45.948881    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l" event={"ID":"b49c2905-88aa-496f-924c-c94e8818dd7f","Type":"ContainerStarted","Data":"e0a842660f8d4794ae3cb57eeeaf4035006ebd0a3989ab10050b3930286428b5"}
Nov 03 13:26:45.949341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:45.948956    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l" event={"ID":"b49c2905-88aa-496f-924c-c94e8818dd7f","Type":"ContainerStarted","Data":"33c53d530338816e9f0dc0d4cb2f21d488548ce479969cb334b57df4722f2113"}
Nov 03 13:26:45.970686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:45.970633    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l" podStartSLOduration=1.970618789 podStartE2EDuration="1.970618789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:45.964727173 +0000 UTC m=+1047.042492193" watchObservedRunningTime="2026-11-03 13:26:45.970618789 +0000 UTC m=+1047.048383815"
Nov 03 13:26:46.009666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.009594    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:46.137809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.137784    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:46.214858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.214826    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-xks-ca\") pod \"15437773-5d9d-4743-ae1b-5b84f43c1daf\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") "
Nov 03 13:26:46.215089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.215076    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-openshift-ca\") pod \"15437773-5d9d-4743-ae1b-5b84f43c1daf\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") "
Nov 03 13:26:46.215140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.215107    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15437773-5d9d-4743-ae1b-5b84f43c1daf-kube-api-access-lb7wz\" (UniqueName: \"kubernetes.io/projected/15437773-5d9d-4743-ae1b-5b84f43c1daf-kube-api-access-lb7wz\") pod \"15437773-5d9d-4743-ae1b-5b84f43c1daf\" (UID: \"15437773-5d9d-4743-ae1b-5b84f43c1daf\") "
Nov 03 13:26:46.215422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.215391    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-openshift-ca" pod "15437773-5d9d-4743-ae1b-5b84f43c1daf" (UID: "15437773-5d9d-4743-ae1b-5b84f43c1daf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:46.215621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.215604    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:46.216886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.216866    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-xks-ca" pod "15437773-5d9d-4743-ae1b-5b84f43c1daf" (UID: "15437773-5d9d-4743-ae1b-5b84f43c1daf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:46.217067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.217044    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15437773-5d9d-4743-ae1b-5b84f43c1daf-kube-api-access-lb7wz" pod "15437773-5d9d-4743-ae1b-5b84f43c1daf" (UID: "15437773-5d9d-4743-ae1b-5b84f43c1daf"). InnerVolumeSpecName "kube-api-access-lb7wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:46.316885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.316853    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15437773-5d9d-4743-ae1b-5b84f43c1daf-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:46.316885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.316887    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb7wz\" (UniqueName: \"kubernetes.io/projected/15437773-5d9d-4743-ae1b-5b84f43c1daf-kube-api-access-lb7wz\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:46.587439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.587408    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:46.959749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.959716    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"
Nov 03 13:26:46.960250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.959713    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj" event={"ID":"15437773-5d9d-4743-ae1b-5b84f43c1daf","Type":"ContainerDied","Data":"98e6c40ad089c6dc902cace06f688d1511bf670bc8017ddaa3e74a4341fd3528"}
Nov 03 13:26:46.960250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.959847    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98e6c40ad089c6dc902cace06f688d1511bf670bc8017ddaa3e74a4341fd3528"
Nov 03 13:26:46.961771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.961743    5420 generic.go:356] "Generic (PLEG): container finished" podID="b49c2905-88aa-496f-924c-c94e8818dd7f" containerID="e0a842660f8d4794ae3cb57eeeaf4035006ebd0a3989ab10050b3930286428b5" exitCode=0
Nov 03 13:26:46.961910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:46.961872    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l" event={"ID":"b49c2905-88aa-496f-924c-c94e8818dd7f","Type":"ContainerDied","Data":"e0a842660f8d4794ae3cb57eeeaf4035006ebd0a3989ab10050b3930286428b5"}
Nov 03 13:26:47.228451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:47.228357    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:47.798690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:47.798662    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:47.990285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:47.990251    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"]
Nov 03 13:26:47.990953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:47.990939    5420 cpu_manager.go:414] "containerMap: removing container" podUID="15437773-5d9d-4743-ae1b-5b84f43c1daf" containerName="revoke-keys"
Nov 03 13:26:47.990995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:47.990954    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15437773-5d9d-4743-ae1b-5b84f43c1daf" containerName="revoke-keys"
Nov 03 13:26:47.991062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:47.991054    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="15437773-5d9d-4743-ae1b-5b84f43c1daf" containerName="revoke-keys"
Nov 03 13:26:47.995020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:47.994997    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.002594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.002571    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-8f8aa8-openshift-default-dockercfg-cvr4j\""
Nov 03 13:26:48.007083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.007059    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"]
Nov 03 13:26:48.145616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.145592    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.145727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.145663    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fbd726-c361-4e4e-8573-f92faefe6de0-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.145800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.145787    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.145847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.145824    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.145889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.145847    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59rh\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-kube-api-access-q59rh\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.145951 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.145932    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.146009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.145996    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.146092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.146039    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.146092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.146060    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.162759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.162738    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:48.246840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.246813    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-openshift-ca\") pod \"b49c2905-88aa-496f-924c-c94e8818dd7f\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") "
Nov 03 13:26:48.247006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.246992    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-xks-ca\") pod \"b49c2905-88aa-496f-924c-c94e8818dd7f\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") "
Nov 03 13:26:48.247113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247093    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49c2905-88aa-496f-924c-c94e8818dd7f-kube-api-access-v7wwr\" (UniqueName: \"kubernetes.io/projected/b49c2905-88aa-496f-924c-c94e8818dd7f-kube-api-access-v7wwr\") pod \"b49c2905-88aa-496f-924c-c94e8818dd7f\" (UID: \"b49c2905-88aa-496f-924c-c94e8818dd7f\") "
Nov 03 13:26:48.247256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247197    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-openshift-ca" pod "b49c2905-88aa-496f-924c-c94e8818dd7f" (UID: "b49c2905-88aa-496f-924c-c94e8818dd7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:48.247454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247431    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.247557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247461    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.247557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247495    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.247687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247578    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fbd726-c361-4e4e-8573-f92faefe6de0-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.247797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247732    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.247797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247791    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.247917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247828    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59rh\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-kube-api-access-q59rh\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.247917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247887    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.247917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.247907    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.248084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.248010    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.248138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.248110    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:48.248246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.248206    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.248356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.248331    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.248432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.248412    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.248490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.248416    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fbd726-c361-4e4e-8573-f92faefe6de0-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.249690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.249667    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-xks-ca" pod "b49c2905-88aa-496f-924c-c94e8818dd7f" (UID: "b49c2905-88aa-496f-924c-c94e8818dd7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:48.249764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.249701    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49c2905-88aa-496f-924c-c94e8818dd7f-kube-api-access-v7wwr" pod "b49c2905-88aa-496f-924c-c94e8818dd7f" (UID: "b49c2905-88aa-496f-924c-c94e8818dd7f"). InnerVolumeSpecName "kube-api-access-v7wwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:48.249764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.249754    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.250280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.250263    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.257439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.257412    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59rh\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-kube-api-access-q59rh\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.257697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.257678    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.315221 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.315188    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:48.349903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.349873    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wwr\" (UniqueName: \"kubernetes.io/projected/b49c2905-88aa-496f-924c-c94e8818dd7f-kube-api-access-v7wwr\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:48.349903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.349903    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c2905-88aa-496f-924c-c94e8818dd7f-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:48.470411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.470378    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:48.616136 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:26:48.616104    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fbd726_c361_4e4e_8573_f92faefe6de0.slice/crio-cefbfabd82d481e82892b517a6de854e7d754445933b910a66ffa8a72d19e4fd WatchSource:0}: Error finding container cefbfabd82d481e82892b517a6de854e7d754445933b910a66ffa8a72d19e4fd: Status 404 returned error can't find the container with id cefbfabd82d481e82892b517a6de854e7d754445933b910a66ffa8a72d19e4fd
Nov 03 13:26:48.617283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.617255    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"]
Nov 03 13:26:48.619298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.619267    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:48.619379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.619329    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:48.619379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.619359    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:26:48.986907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.986861    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" event={"ID":"85fbd726-c361-4e4e-8573-f92faefe6de0","Type":"ContainerStarted","Data":"7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d"}
Nov 03 13:26:48.987100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.986934    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" event={"ID":"85fbd726-c361-4e4e-8573-f92faefe6de0","Type":"ContainerStarted","Data":"cefbfabd82d481e82892b517a6de854e7d754445933b910a66ffa8a72d19e4fd"}
Nov 03 13:26:48.988759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.988735    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"
Nov 03 13:26:48.988911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.988737    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l" event={"ID":"b49c2905-88aa-496f-924c-c94e8818dd7f","Type":"ContainerDied","Data":"33c53d530338816e9f0dc0d4cb2f21d488548ce479969cb334b57df4722f2113"}
Nov 03 13:26:48.988911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:48.988864    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33c53d530338816e9f0dc0d4cb2f21d488548ce479969cb334b57df4722f2113"
Nov 03 13:26:49.019578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:49.019524    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" podStartSLOduration=2.019506811 podStartE2EDuration="2.019506811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:49.014828136 +0000 UTC m=+1050.092593162" watchObservedRunningTime="2026-11-03 13:26:49.019506811 +0000 UTC m=+1050.097271836"
Nov 03 13:26:49.053761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:49.053721    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:49.315616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:49.315532    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:49.317042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:49.317018    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 13:26:49.317148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:49.317068    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 13:26:49.876142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:49.873165    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:50.315612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:50.315580    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 13:26:50.316147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:50.315637    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 13:26:50.566288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:50.565470    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:51.315588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:51.315551    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 13:26:51.315766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:51.315611    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 13:26:51.458945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:51.458901    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:52.163235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.158281    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:52.315462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.315420    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 13:26:52.315628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.315474    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 13:26:52.722263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.721045    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"]
Nov 03 13:26:52.747264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.747203    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b49c2905-88aa-496f-924c-c94e8818dd7f" containerName="revoke-keys"
Nov 03 13:26:52.749303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.749276    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49c2905-88aa-496f-924c-c94e8818dd7f" containerName="revoke-keys"
Nov 03 13:26:52.749680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.749662    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49c2905-88aa-496f-924c-c94e8818dd7f" containerName="revoke-keys"
Nov 03 13:26:52.754235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.754201    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:52.756156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.756127    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"]
Nov 03 13:26:52.758985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.758948    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:26:52.759168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.759137    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:26:52.759288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.759269    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:26:52.759427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.759409    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:26:52.813197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.813170    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-380ac2d7-twj6q\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:52.813349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.813331    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchln\" (UniqueName: \"kubernetes.io/projected/be070a20-aa77-448a-a06d-93c631c036f3-kube-api-access-hchln\") pod \"maas-api-revoke-sub-380ac2d7-twj6q\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:52.813509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.813495    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-380ac2d7-twj6q\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:52.914853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.914822    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchln\" (UniqueName: \"kubernetes.io/projected/be070a20-aa77-448a-a06d-93c631c036f3-kube-api-access-hchln\") pod \"maas-api-revoke-sub-380ac2d7-twj6q\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:52.915011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.914985    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-380ac2d7-twj6q\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:52.915182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.915165    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-380ac2d7-twj6q\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:52.915631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.915612    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-380ac2d7-twj6q\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:52.917491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.917470    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-380ac2d7-twj6q\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:52.947911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:52.947859    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchln\" (UniqueName: \"kubernetes.io/projected/be070a20-aa77-448a-a06d-93c631c036f3-kube-api-access-hchln\") pod \"maas-api-revoke-sub-380ac2d7-twj6q\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:53.076096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:53.076019    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:53.082048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:53.081970    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:53.315542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:53.315511    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 13:26:53.315733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:53.315564    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 13:26:53.413187 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:26:53.413160    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe070a20_aa77_448a_a06d_93c631c036f3.slice/crio-937d676deb39f645df642e7e0ab975ca293f096a5af3711f427ee2a124267b67 WatchSource:0}: Error finding container 937d676deb39f645df642e7e0ab975ca293f096a5af3711f427ee2a124267b67: Status 404 returned error can't find the container with id 937d676deb39f645df642e7e0ab975ca293f096a5af3711f427ee2a124267b67
Nov 03 13:26:53.414307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:53.414285    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"]
Nov 03 13:26:53.801617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:53.801579    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:54.061248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:54.061206    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q" event={"ID":"be070a20-aa77-448a-a06d-93c631c036f3","Type":"ContainerStarted","Data":"14705518bd7fe68e582be63e8a3ada05b208f0bcb32372094feb84062788f880"}
Nov 03 13:26:54.061417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:54.061266    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q" event={"ID":"be070a20-aa77-448a-a06d-93c631c036f3","Type":"ContainerStarted","Data":"937d676deb39f645df642e7e0ab975ca293f096a5af3711f427ee2a124267b67"}
Nov 03 13:26:54.082724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:54.082677    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q" podStartSLOduration=2.082663858 podStartE2EDuration="2.082663858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:54.076715981 +0000 UTC m=+1055.154481000" watchObservedRunningTime="2026-11-03 13:26:54.082663858 +0000 UTC m=+1055.160428876"
Nov 03 13:26:54.315910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:54.315873    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 13:26:54.316083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:54.315930    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 13:26:54.445233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:54.434270    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:54.732126 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:26:54.732086    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:26:54.919540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:54.919454    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:55.073733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:55.073702    5420 generic.go:356] "Generic (PLEG): container finished" podID="be070a20-aa77-448a-a06d-93c631c036f3" containerID="14705518bd7fe68e582be63e8a3ada05b208f0bcb32372094feb84062788f880" exitCode=0
Nov 03 13:26:55.073949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:55.073795    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q" event={"ID":"be070a20-aa77-448a-a06d-93c631c036f3","Type":"ContainerDied","Data":"14705518bd7fe68e582be63e8a3ada05b208f0bcb32372094feb84062788f880"}
Nov 03 13:26:55.082605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:55.082582    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:55.217435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:55.217401    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:55.323776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:55.323745    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:55.324277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:55.324251    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:55.325191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:55.325168    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:26:55.793578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:55.792223    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:56.240398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.240372    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:56.309801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.309768    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-openshift-ca\") pod \"be070a20-aa77-448a-a06d-93c631c036f3\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") "
Nov 03 13:26:56.309961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.309832    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be070a20-aa77-448a-a06d-93c631c036f3-kube-api-access-hchln\" (UniqueName: \"kubernetes.io/projected/be070a20-aa77-448a-a06d-93c631c036f3-kube-api-access-hchln\") pod \"be070a20-aa77-448a-a06d-93c631c036f3\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") "
Nov 03 13:26:56.310078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.310048    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-xks-ca\") pod \"be070a20-aa77-448a-a06d-93c631c036f3\" (UID: \"be070a20-aa77-448a-a06d-93c631c036f3\") "
Nov 03 13:26:56.310190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.310165    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-openshift-ca" pod "be070a20-aa77-448a-a06d-93c631c036f3" (UID: "be070a20-aa77-448a-a06d-93c631c036f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:26:56.310770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.310745    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:56.312346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.312190    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-xks-ca" pod "be070a20-aa77-448a-a06d-93c631c036f3" (UID: "be070a20-aa77-448a-a06d-93c631c036f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:26:56.312806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.312782    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be070a20-aa77-448a-a06d-93c631c036f3-kube-api-access-hchln" pod "be070a20-aa77-448a-a06d-93c631c036f3" (UID: "be070a20-aa77-448a-a06d-93c631c036f3"). InnerVolumeSpecName "kube-api-access-hchln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:26:56.413143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.413065    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be070a20-aa77-448a-a06d-93c631c036f3-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:56.413143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.413091    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchln\" (UniqueName: \"kubernetes.io/projected/be070a20-aa77-448a-a06d-93c631c036f3-kube-api-access-hchln\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:26:56.553637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:56.553594    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:57.053206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.053176    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:57.096512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.096348    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"
Nov 03 13:26:57.096607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.096372    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q" event={"ID":"be070a20-aa77-448a-a06d-93c631c036f3","Type":"ContainerDied","Data":"937d676deb39f645df642e7e0ab975ca293f096a5af3711f427ee2a124267b67"}
Nov 03 13:26:57.096607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.096587    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="937d676deb39f645df642e7e0ab975ca293f096a5af3711f427ee2a124267b67"
Nov 03 13:26:57.243087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.243052    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:57.311825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.311733    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"]
Nov 03 13:26:57.318305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.318276    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e04a4e0c-qsv5l"]
Nov 03 13:26:57.392636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.392601    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:26:57.414234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.405056    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:57.581802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:57.581726    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:58.348311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:58.348282    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:59.175674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:59.175642    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:59.214135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:59.208935    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49c2905-88aa-496f-924c-c94e8818dd7f" path="/var/lib/kubelet/pods/b49c2905-88aa-496f-924c-c94e8818dd7f/volumes"
Nov 03 13:26:59.214135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:59.209663    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"]
Nov 03 13:26:59.214135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:59.210871    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4" podUID="ccc2a373-b468-4904-9dff-770419fc4e2a" containerName="istio-proxy" containerID="cri-o://0c23c8c70506a936284bacc82dc39964ee3e413d80a35857c0877888cd85b9d2" gracePeriod=30
Nov 03 13:26:59.365846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:59.365785    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:59.677910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:26:59.677878    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:00.057232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:00.052804    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:00.505959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:00.505916    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:00.819768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:00.819625    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:01.043637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:01.043605    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:01.924757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:01.924724    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:03.006712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:03.006676    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:03.689785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:03.689748    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:03.870230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:03.862022    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:04.575678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:04.575471    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:05.006835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:05.006800    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:05.172267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:05.172183    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:05.565944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:05.565900    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:06.338719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:06.338683    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:07.014313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.013184    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:07.138734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.138706    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"]
Nov 03 13:27:07.139410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.139396    5420 cpu_manager.go:414] "containerMap: removing container" podUID="be070a20-aa77-448a-a06d-93c631c036f3" containerName="revoke-keys"
Nov 03 13:27:07.139454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.139411    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be070a20-aa77-448a-a06d-93c631c036f3" containerName="revoke-keys"
Nov 03 13:27:07.139542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.139533    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="be070a20-aa77-448a-a06d-93c631c036f3" containerName="revoke-keys"
Nov 03 13:27:07.143476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.143461    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.152562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.152537    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"]
Nov 03 13:27:07.162813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.162795    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:27:07.163002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.162986    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:27:07.163084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.163061    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:27:07.163167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.163125    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:27:07.196270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.196236    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxfmr\" (UniqueName: \"kubernetes.io/projected/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-kube-api-access-jxfmr\") pod \"maas-api-revoke-sub-935084bc-x4sb9\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.196448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.196414    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-935084bc-x4sb9\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.196572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.196551    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-935084bc-x4sb9\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.297808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.297729    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxfmr\" (UniqueName: \"kubernetes.io/projected/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-kube-api-access-jxfmr\") pod \"maas-api-revoke-sub-935084bc-x4sb9\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.297978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.297853    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-935084bc-x4sb9\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.297978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.297944    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-935084bc-x4sb9\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.298614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.298590    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-935084bc-x4sb9\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.300199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.300179    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-935084bc-x4sb9\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.316019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.315992    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxfmr\" (UniqueName: \"kubernetes.io/projected/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-kube-api-access-jxfmr\") pod \"maas-api-revoke-sub-935084bc-x4sb9\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.470427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.470393    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:07.761237 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:27:07.760289    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ee7256_cdfa_4c2f_99bc_e1f6c2602910.slice/crio-8d72adfb697f8edce53a3d0f71c38a1bb2b2e4d1241c76f75024d7d407cc222b WatchSource:0}: Error finding container 8d72adfb697f8edce53a3d0f71c38a1bb2b2e4d1241c76f75024d7d407cc222b: Status 404 returned error can't find the container with id 8d72adfb697f8edce53a3d0f71c38a1bb2b2e4d1241c76f75024d7d407cc222b
Nov 03 13:27:07.764104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.764080    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"]
Nov 03 13:27:07.816882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:07.816851    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:08.417983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.415452    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9" event={"ID":"45ee7256-cdfa-4c2f-99bc-e1f6c2602910","Type":"ContainerStarted","Data":"6c9df22a1d988f2c9f5b108f725e7b945c151a441417c02b207f4a576dfaa1fe"}
Nov 03 13:27:08.417983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.415511    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9" event={"ID":"45ee7256-cdfa-4c2f-99bc-e1f6c2602910","Type":"ContainerStarted","Data":"8d72adfb697f8edce53a3d0f71c38a1bb2b2e4d1241c76f75024d7d407cc222b"}
Nov 03 13:27:08.418289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.418183    5420 generic.go:356] "Generic (PLEG): container finished" podID="ccc2a373-b468-4904-9dff-770419fc4e2a" containerID="0c23c8c70506a936284bacc82dc39964ee3e413d80a35857c0877888cd85b9d2" exitCode=0
Nov 03 13:27:08.418366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.418353    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4" event={"ID":"ccc2a373-b468-4904-9dff-770419fc4e2a","Type":"ContainerDied","Data":"0c23c8c70506a936284bacc82dc39964ee3e413d80a35857c0877888cd85b9d2"}
Nov 03 13:27:08.443016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.442962    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9" podStartSLOduration=1.442944727 podStartE2EDuration="1.442944727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:08.437537943 +0000 UTC m=+1069.515302970" watchObservedRunningTime="2026-11-03 13:27:08.442944727 +0000 UTC m=+1069.520709757"
Nov 03 13:27:08.490600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.490564    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:08.517060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.517031    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:27:08.620727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.620644    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc2a373-b468-4904-9dff-770419fc4e2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2a373-b468-4904-9dff-770419fc4e2a-istiod-ca-cert\") pod \"ccc2a373-b468-4904-9dff-770419fc4e2a\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") "
Nov 03 13:27:08.620727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.620680    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-kube-api-access-zcl4v\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-kube-api-access-zcl4v\") pod \"ccc2a373-b468-4904-9dff-770419fc4e2a\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") "
Nov 03 13:27:08.620727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.620710    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-envoy\") pod \"ccc2a373-b468-4904-9dff-770419fc4e2a\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") "
Nov 03 13:27:08.621394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.620736    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-data\") pod \"ccc2a373-b468-4904-9dff-770419fc4e2a\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") "
Nov 03 13:27:08.621394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.620783    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-podinfo\") pod \"ccc2a373-b468-4904-9dff-770419fc4e2a\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") "
Nov 03 13:27:08.621394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.620847    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-token\") pod \"ccc2a373-b468-4904-9dff-770419fc4e2a\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") "
Nov 03 13:27:08.621394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.620884    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-credential-socket\") pod \"ccc2a373-b468-4904-9dff-770419fc4e2a\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") "
Nov 03 13:27:08.621394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621027    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-socket\") pod \"ccc2a373-b468-4904-9dff-770419fc4e2a\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") "
Nov 03 13:27:08.621394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621118    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-certs\") pod \"ccc2a373-b468-4904-9dff-770419fc4e2a\" (UID: \"ccc2a373-b468-4904-9dff-770419fc4e2a\") "
Nov 03 13:27:08.621394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621188    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-data" pod "ccc2a373-b468-4904-9dff-770419fc4e2a" (UID: "ccc2a373-b468-4904-9dff-770419fc4e2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:08.621394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621204    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-credential-socket" pod "ccc2a373-b468-4904-9dff-770419fc4e2a" (UID: "ccc2a373-b468-4904-9dff-770419fc4e2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:08.621394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621252    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc2a373-b468-4904-9dff-770419fc4e2a-istiod-ca-cert" pod "ccc2a373-b468-4904-9dff-770419fc4e2a" (UID: "ccc2a373-b468-4904-9dff-770419fc4e2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:08.621917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621403    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-socket" pod "ccc2a373-b468-4904-9dff-770419fc4e2a" (UID: "ccc2a373-b468-4904-9dff-770419fc4e2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:08.621917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621497    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-certs" pod "ccc2a373-b468-4904-9dff-770419fc4e2a" (UID: "ccc2a373-b468-4904-9dff-770419fc4e2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:08.622044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621925    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:08.622044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621949    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:08.622044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621964    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2a373-b468-4904-9dff-770419fc4e2a-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:08.622044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621980    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:08.622044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.621995    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:08.623266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.623236    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-envoy" pod "ccc2a373-b468-4904-9dff-770419fc4e2a" (UID: "ccc2a373-b468-4904-9dff-770419fc4e2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:27:08.623516 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.623484    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-kube-api-access-zcl4v" pod "ccc2a373-b468-4904-9dff-770419fc4e2a" (UID: "ccc2a373-b468-4904-9dff-770419fc4e2a"). InnerVolumeSpecName "kube-api-access-zcl4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:08.623613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.623551    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-token" pod "ccc2a373-b468-4904-9dff-770419fc4e2a" (UID: "ccc2a373-b468-4904-9dff-770419fc4e2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:08.623654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.623645    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-podinfo" pod "ccc2a373-b468-4904-9dff-770419fc4e2a" (UID: "ccc2a373-b468-4904-9dff-770419fc4e2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:27:08.723701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.723674    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcl4v\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-kube-api-access-zcl4v\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:08.723701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.723700    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:08.723893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.723711    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:08.723893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:08.723721    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2a373-b468-4904-9dff-770419fc4e2a-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:09.098424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:09.098381    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:09.428997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:09.428915    5420 generic.go:356] "Generic (PLEG): container finished" podID="45ee7256-cdfa-4c2f-99bc-e1f6c2602910" containerID="6c9df22a1d988f2c9f5b108f725e7b945c151a441417c02b207f4a576dfaa1fe" exitCode=0
Nov 03 13:27:09.429148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:09.429004    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9" event={"ID":"45ee7256-cdfa-4c2f-99bc-e1f6c2602910","Type":"ContainerDied","Data":"6c9df22a1d988f2c9f5b108f725e7b945c151a441417c02b207f4a576dfaa1fe"}
Nov 03 13:27:09.430690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:09.430667    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"
Nov 03 13:27:09.430829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:09.430701    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4" event={"ID":"ccc2a373-b468-4904-9dff-770419fc4e2a","Type":"ContainerDied","Data":"7ff639eb47828f80ce9f9866cef4e1c6ecd5617e02c2633644d5f109908d46cb"}
Nov 03 13:27:09.430829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:09.430760    5420 scope.go:122] "RemoveContainer" containerID="0c23c8c70506a936284bacc82dc39964ee3e413d80a35857c0877888cd85b9d2"
Nov 03 13:27:09.468992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:09.468964    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"]
Nov 03 13:27:09.474619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:09.474596    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e04a4e0c-openshift-default-75674dbfb9-l6js4"]
Nov 03 13:27:09.703280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:09.702923    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:10.339132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.339102    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:10.623412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.623384    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:10.754311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.754284    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-xks-ca\") pod \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") "
Nov 03 13:27:10.754664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.754352    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-kube-api-access-jxfmr\" (UniqueName: \"kubernetes.io/projected/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-kube-api-access-jxfmr\") pod \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") "
Nov 03 13:27:10.754664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.754496    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-openshift-ca\") pod \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\" (UID: \"45ee7256-cdfa-4c2f-99bc-e1f6c2602910\") "
Nov 03 13:27:10.754886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.754851    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-openshift-ca" pod "45ee7256-cdfa-4c2f-99bc-e1f6c2602910" (UID: "45ee7256-cdfa-4c2f-99bc-e1f6c2602910"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:10.755034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.755018    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:10.756343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.756318    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-kube-api-access-jxfmr" pod "45ee7256-cdfa-4c2f-99bc-e1f6c2602910" (UID: "45ee7256-cdfa-4c2f-99bc-e1f6c2602910"). InnerVolumeSpecName "kube-api-access-jxfmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:10.756452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.756388    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-xks-ca" pod "45ee7256-cdfa-4c2f-99bc-e1f6c2602910" (UID: "45ee7256-cdfa-4c2f-99bc-e1f6c2602910"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:27:10.856862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.856803    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:10.856862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:10.856828    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxfmr\" (UniqueName: \"kubernetes.io/projected/45ee7256-cdfa-4c2f-99bc-e1f6c2602910-kube-api-access-jxfmr\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:11.086339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:11.086309    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc2a373-b468-4904-9dff-770419fc4e2a" path="/var/lib/kubelet/pods/ccc2a373-b468-4904-9dff-770419fc4e2a/volumes"
Nov 03 13:27:11.117527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:11.117496    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:11.269401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:11.269368    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:11.425890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:11.425813    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:11.454393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:11.454365    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9" event={"ID":"45ee7256-cdfa-4c2f-99bc-e1f6c2602910","Type":"ContainerDied","Data":"8d72adfb697f8edce53a3d0f71c38a1bb2b2e4d1241c76f75024d7d407cc222b"}
Nov 03 13:27:11.454535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:11.454398    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"
Nov 03 13:27:11.454535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:11.454418    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d72adfb697f8edce53a3d0f71c38a1bb2b2e4d1241c76f75024d7d407cc222b"
Nov 03 13:27:11.718974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:11.718936    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:12.096312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:12.096207    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:12.701675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:12.701644    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:13.202508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.202481    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"]
Nov 03 13:27:13.203191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.203174    5420 cpu_manager.go:414] "containerMap: removing container" podUID="ccc2a373-b468-4904-9dff-770419fc4e2a" containerName="istio-proxy"
Nov 03 13:27:13.203276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.203193    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc2a373-b468-4904-9dff-770419fc4e2a" containerName="istio-proxy"
Nov 03 13:27:13.203276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.203249    5420 cpu_manager.go:414] "containerMap: removing container" podUID="45ee7256-cdfa-4c2f-99bc-e1f6c2602910" containerName="revoke-keys"
Nov 03 13:27:13.203276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.203256    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ee7256-cdfa-4c2f-99bc-e1f6c2602910" containerName="revoke-keys"
Nov 03 13:27:13.203372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.203355    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ee7256-cdfa-4c2f-99bc-e1f6c2602910" containerName="revoke-keys"
Nov 03 13:27:13.203372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.203370    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc2a373-b468-4904-9dff-770419fc4e2a" containerName="istio-proxy"
Nov 03 13:27:13.206851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.206832    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.233718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.233695    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"]
Nov 03 13:27:13.262780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.262751    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"]
Nov 03 13:27:13.267028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.267005    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.287327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.287303    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"]
Nov 03 13:27:13.390070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.390035    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:13.401030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.400996    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.401381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401341    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.401514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401395    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.401514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401478    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.401630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401512    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.401630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401537    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.401630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401586    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.401778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401627    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.401778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401669    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.401778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401705    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c74d124-2b67-4752-9725-ef7146010303-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.401938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401779    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.401938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401820    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.401938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401901    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c74d124-2b67-4752-9725-ef7146010303-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.401938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401928    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b889q\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-kube-api-access-b889q\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.402137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.401976    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.402137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.402017    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.402137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.402071    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.402317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.402150    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xg9k\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-kube-api-access-8xg9k\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.504079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.503995    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.504079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504038    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.504370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504083    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.504370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504262    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xg9k\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-kube-api-access-8xg9k\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.504506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504379    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.504506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504427    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.504506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504453    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.504506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504474    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.504506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504486    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.504772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504527    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.504772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504598    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.504772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504634    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.504772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504661    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.504772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504709    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.504772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504750    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.505065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504791    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.505065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504825    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c74d124-2b67-4752-9725-ef7146010303-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.505065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504886    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.505065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504901    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.505065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504933    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.505065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.504944    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.505688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.505663    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.505769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.505674    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.505961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.505939    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.506114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.506092    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c74d124-2b67-4752-9725-ef7146010303-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.506187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.506144    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b889q\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-kube-api-access-b889q\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.506390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.506363    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.509538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.506702    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c74d124-2b67-4752-9725-ef7146010303-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.509655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.508557    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.509655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.508722    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.509655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.508771    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.509818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.509650    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c74d124-2b67-4752-9725-ef7146010303-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.520198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.513843    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.520198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.516169    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xg9k\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-kube-api-access-8xg9k\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.522438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.522417    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b889q\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-kube-api-access-b889q\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.522525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.522440    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.585094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.585067    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:13.821291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.821171    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:13.878811 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:27:13.878783    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1e78d4_0bff_4405_9e3a_a11d0835a719.slice/crio-0404c620f83f8cbe47270ea4a011bd30d06c3308746b98490c447eb90a9463f4 WatchSource:0}: Error finding container 0404c620f83f8cbe47270ea4a011bd30d06c3308746b98490c447eb90a9463f4: Status 404 returned error can't find the container with id 0404c620f83f8cbe47270ea4a011bd30d06c3308746b98490c447eb90a9463f4
Nov 03 13:27:13.881619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.881584    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:27:13.881711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.881653    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:27:13.881711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.881698    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:27:13.882746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:13.882450    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"]
Nov 03 13:27:14.032840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.032693    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:14.123105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.123072    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"]
Nov 03 13:27:14.123634 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:27:14.123610    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c74d124_2b67_4752_9725_ef7146010303.slice/crio-1624026e3a17578ddef1fffe18d121464a9354c6c8c1ca6e8a3bd052d52bd2e9 WatchSource:0}: Error finding container 1624026e3a17578ddef1fffe18d121464a9354c6c8c1ca6e8a3bd052d52bd2e9: Status 404 returned error can't find the container with id 1624026e3a17578ddef1fffe18d121464a9354c6c8c1ca6e8a3bd052d52bd2e9
Nov 03 13:27:14.125762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.125728    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:27:14.125844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.125793    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:27:14.125844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.125823    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:27:14.484837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.484801    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" event={"ID":"eb1e78d4-0bff-4405-9e3a-a11d0835a719","Type":"ContainerStarted","Data":"08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1"}
Nov 03 13:27:14.485286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.484858    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" event={"ID":"eb1e78d4-0bff-4405-9e3a-a11d0835a719","Type":"ContainerStarted","Data":"0404c620f83f8cbe47270ea4a011bd30d06c3308746b98490c447eb90a9463f4"}
Nov 03 13:27:14.486740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.486712    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" event={"ID":"2c74d124-2b67-4752-9725-ef7146010303","Type":"ContainerStarted","Data":"a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0"}
Nov 03 13:27:14.486841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.486759    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" event={"ID":"2c74d124-2b67-4752-9725-ef7146010303","Type":"ContainerStarted","Data":"1624026e3a17578ddef1fffe18d121464a9354c6c8c1ca6e8a3bd052d52bd2e9"}
Nov 03 13:27:14.512970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.512918    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" podStartSLOduration=1.5129001930000001 podStartE2EDuration="1.512900193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:14.507058589 +0000 UTC m=+1075.584823616" watchObservedRunningTime="2026-11-03 13:27:14.512900193 +0000 UTC m=+1075.590665222"
Nov 03 13:27:14.531270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.531223    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" podStartSLOduration=1.531194416 podStartE2EDuration="1.531194416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:14.527443325 +0000 UTC m=+1075.605208354" watchObservedRunningTime="2026-11-03 13:27:14.531194416 +0000 UTC m=+1075.608959441"
Nov 03 13:27:14.585788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.585758    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:14.587725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.587694    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 13:27:14.587875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.587742    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 13:27:14.708885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.708849    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:14.832307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:14.831323    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:15.399745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:15.399705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:15.586141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:15.586111    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 13:27:15.586548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:15.586170    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 13:27:15.829237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:15.827163    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 13:27:15.833232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:15.832257    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 13:27:16.118619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:16.118533    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:16.599278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:16.589261    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 13:27:16.599278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:16.589325    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 13:27:16.853343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:16.837276    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 13:27:16.853343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:16.837336    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 13:27:16.915420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:16.910090    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:17.140565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:17.140477    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:17.792235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:17.786905    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 13:27:17.792235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:17.786961    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 13:27:17.848655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:17.837935    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 13:27:17.848655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:17.838008    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 13:27:17.988264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:17.982987    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:18.284988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:18.284931    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded"
Nov 03 13:27:18.328241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:18.327283    5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv" podUID="a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:27:18.574297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:18.570714    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:18.589035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:18.588563    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 13:27:18.589035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:18.588636    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 13:27:18.871403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:18.860183    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 13:27:18.871403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:18.869334    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 13:27:19.087116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:19.087091    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"]
Nov 03 13:27:19.087116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:19.087126    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1641ad3-zmb7c"]
Nov 03 13:27:19.570235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:19.569738    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:19.587234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:19.586557    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 13:27:19.587234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:19.586617    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 13:27:19.771026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:19.770982    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:19.846286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:19.842550    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 13:27:19.846286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:19.842609    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 13:27:20.378170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.378133    5420 scope.go:122] "RemoveContainer" containerID="21181cd12e1744d7ae071240bb4831a661e1fccce4e966f70a0722f22a08256a"
Nov 03 13:27:20.391145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.391121    5420 scope.go:122] "RemoveContainer" containerID="08dd67dc1ff42757087f318069db624140b7b338c371876d9dab4c7322179a9e"
Nov 03 13:27:20.407396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.407374    5420 scope.go:122] "RemoveContainer" containerID="b393ef507abc25cc52123b541e2e60c741b2b0dd419ddc950efd3daecdf284d5"
Nov 03 13:27:20.454231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.446433    5420 scope.go:122] "RemoveContainer" containerID="25007adaafed1af6ab03d19977f4dab783d3a6e932076724076bc5c283fee3aa"
Nov 03 13:27:20.454231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.449486    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:20.484098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.484070    5420 scope.go:122] "RemoveContainer" containerID="93eae85c2dec99191d7994ec4469440f9c48d7244c4427c62f3848cdf527b00d"
Nov 03 13:27:20.499978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.499837    5420 scope.go:122] "RemoveContainer" containerID="8260816a0f45e5bbbbbfc27518d4048c52d04f4d6f0cf9dfd1136f0797a03417"
Nov 03 13:27:20.513111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.513089    5420 scope.go:122] "RemoveContainer" containerID="d8b37c8c3eebd9a38faa88d3158afa6e8bd987d313c3f3cad316e6e4d9884d4a"
Nov 03 13:27:20.555809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.555709    5420 scope.go:122] "RemoveContainer" containerID="03ef318fe778be99da0605bb5eb908dc9e59949848b33bf04b6b10acb137cb46"
Nov 03 13:27:20.567719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.567703    5420 scope.go:122] "RemoveContainer" containerID="d65c7e3b6bc3f7d3b0a1b98a24d541e66d7084639177115b2bcc1fd5c93b8503"
Nov 03 13:27:20.577775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.577755    5420 scope.go:122] "RemoveContainer" containerID="1adc352a42ce16bec0a269c23169363b6744a84082fe1ba7c1ee5e34ee544a69"
Nov 03 13:27:20.586113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.586090    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 13:27:20.586222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.586144    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 13:27:20.586946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.586926    5420 scope.go:122] "RemoveContainer" containerID="73b916646ef3a4a99812a4a57891c1d1e9ef47cbb883978bf83547f6eb9fe4f3"
Nov 03 13:27:20.596696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.596676    5420 scope.go:122] "RemoveContainer" containerID="dbe2513bf4cdf9672f9a69fba158c170a00cbb0d421f6546bd2a06ef235bbd05"
Nov 03 13:27:20.821803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.821769    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 13:27:20.821981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:20.821833    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 13:27:21.088034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:21.087987    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f159b56b-47a7-4352-ae6d-459b9402aa93" path="/var/lib/kubelet/pods/f159b56b-47a7-4352-ae6d-459b9402aa93/volumes"
Nov 03 13:27:21.391705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:21.391628    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:21.589870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:21.589843    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:21.590347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:21.590331    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:21.591146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:21.591125    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:27:21.832366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:21.832339    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:21.833027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:21.832995    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:21.833868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:21.833849    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:27:22.144040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.143275    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"]
Nov 03 13:27:22.150399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.150368    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"]
Nov 03 13:27:22.150781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.150759    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.157604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.157577    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:27:22.157734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.157722    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:27:22.157872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.157855    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:27:22.157957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.157934    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:27:22.207717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.207685    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2709f34-5x4dv\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.207913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.207740    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2709f34-5x4dv\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.207992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.207968    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvgx\" (UniqueName: \"kubernetes.io/projected/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-kube-api-access-8lvgx\") pod \"maas-api-revoke-sub-a2709f34-5x4dv\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.312235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.309761    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2709f34-5x4dv\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.312235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.309826    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2709f34-5x4dv\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.312235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.310008    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvgx\" (UniqueName: \"kubernetes.io/projected/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-kube-api-access-8lvgx\") pod \"maas-api-revoke-sub-a2709f34-5x4dv\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.312235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.311609    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2709f34-5x4dv\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.320089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.315398    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2709f34-5x4dv\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.320089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.317468    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:22.321526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.320714    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvgx\" (UniqueName: \"kubernetes.io/projected/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-kube-api-access-8lvgx\") pod \"maas-api-revoke-sub-a2709f34-5x4dv\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.471887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.471857    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:22.771725 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:27:22.771695    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd5d8dd_8f5f_4a96_acc0_75b62ff1766c.slice/crio-57a0e6061a929933ed6af3ea89aa4da80884381f098eaf56e8fb69a61ac8ae2c WatchSource:0}: Error finding container 57a0e6061a929933ed6af3ea89aa4da80884381f098eaf56e8fb69a61ac8ae2c: Status 404 returned error can't find the container with id 57a0e6061a929933ed6af3ea89aa4da80884381f098eaf56e8fb69a61ac8ae2c
Nov 03 13:27:22.772462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.772436    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"]
Nov 03 13:27:22.848801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:22.848768    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv" event={"ID":"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c","Type":"ContainerStarted","Data":"57a0e6061a929933ed6af3ea89aa4da80884381f098eaf56e8fb69a61ac8ae2c"}
Nov 03 13:27:23.014798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:23.014766    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:23.776147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:23.776110    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:23.859617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:23.859590    5420 generic.go:356] "Generic (PLEG): container finished" podID="4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c" containerID="769b1cfde4ed98d82c02291f133ddc1b36593d74090a2ad21e8385b9dfc85b57" exitCode=0
Nov 03 13:27:23.859756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:23.859661    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv" event={"ID":"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c","Type":"ContainerDied","Data":"769b1cfde4ed98d82c02291f133ddc1b36593d74090a2ad21e8385b9dfc85b57"}
Nov 03 13:27:24.431126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.431080    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:24.694400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.694329    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"]
Nov 03 13:27:24.697870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.697855    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:24.710464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.710423    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"]
Nov 03 13:27:24.744758 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:27:24.744729    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:27:24.748675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.748648    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c87701ad-ftls7\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:24.748908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.748884    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c87701ad-ftls7\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:24.749037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.748919    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs57j\" (UniqueName: \"kubernetes.io/projected/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-kube-api-access-gs57j\") pod \"maas-api-revoke-keys-e2e-ait-c87701ad-ftls7\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:24.851354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.851326    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c87701ad-ftls7\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:24.851810 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.851474    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c87701ad-ftls7\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:24.851810 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.851495    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs57j\" (UniqueName: \"kubernetes.io/projected/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-kube-api-access-gs57j\") pod \"maas-api-revoke-keys-e2e-ait-c87701ad-ftls7\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:24.852144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.852123    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c87701ad-ftls7\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:24.853887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.853859    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c87701ad-ftls7\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:24.861382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:24.861348    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs57j\" (UniqueName: \"kubernetes.io/projected/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-kube-api-access-gs57j\") pod \"maas-api-revoke-keys-e2e-ait-c87701ad-ftls7\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:25.015360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.015318    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:25.034106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.034078    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"]
Nov 03 13:27:25.044650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.044626    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c28f80a-fwvlx"]
Nov 03 13:27:25.048484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.048464    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:25.092514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.092480    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63591942-3ae9-49bd-ae74-293ef518d990" path="/var/lib/kubelet/pods/63591942-3ae9-49bd-ae74-293ef518d990/volumes"
Nov 03 13:27:25.154950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.154923    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-xks-ca\") pod \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") "
Nov 03 13:27:25.155314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.155293    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-openshift-ca\") pod \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") "
Nov 03 13:27:25.155419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.155345    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-kube-api-access-8lvgx\" (UniqueName: \"kubernetes.io/projected/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-kube-api-access-8lvgx\") pod \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\" (UID: \"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c\") "
Nov 03 13:27:25.155662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.155638    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-openshift-ca" pod "4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c" (UID: "4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:25.156236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.156203    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:25.157605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.157579    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-xks-ca" pod "4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c" (UID: "4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:27:25.158298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.158278    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-kube-api-access-8lvgx" pod "4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c" (UID: "4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c"). InnerVolumeSpecName "kube-api-access-8lvgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:25.258244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.258145    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lvgx\" (UniqueName: \"kubernetes.io/projected/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-kube-api-access-8lvgx\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:25.258244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.258173    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:25.300925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.300297    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:25.330640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.329834    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"]
Nov 03 13:27:25.880979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.880955    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"
Nov 03 13:27:25.881407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.880992    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv" event={"ID":"4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c","Type":"ContainerDied","Data":"57a0e6061a929933ed6af3ea89aa4da80884381f098eaf56e8fb69a61ac8ae2c"}
Nov 03 13:27:25.881407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.881041    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57a0e6061a929933ed6af3ea89aa4da80884381f098eaf56e8fb69a61ac8ae2c"
Nov 03 13:27:25.882805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.882780    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7" event={"ID":"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51","Type":"ContainerStarted","Data":"74ffff3291d803106b58f56b32f923e83a7210b4eeab25714ceae665f3ce4d37"}
Nov 03 13:27:25.882936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.882831    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7" event={"ID":"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51","Type":"ContainerStarted","Data":"d8da2c11c6ec0708747e9238d68441d20b84c1e21c1444a16798436166ddffb2"}
Nov 03 13:27:25.914265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:25.914191    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7" podStartSLOduration=1.914178001 podStartE2EDuration="1.914178001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:25.911325283 +0000 UTC m=+1086.989090301" watchObservedRunningTime="2026-11-03 13:27:25.914178001 +0000 UTC m=+1086.991943026"
Nov 03 13:27:26.048359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.048328    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:26.502303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.502275    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"]
Nov 03 13:27:26.502967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.502953    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c" containerName="revoke-keys"
Nov 03 13:27:26.503030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.502968    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c" containerName="revoke-keys"
Nov 03 13:27:26.503127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.503118    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c" containerName="revoke-keys"
Nov 03 13:27:26.505844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.505826    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.520230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.520190    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"]
Nov 03 13:27:26.577135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.577103    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5240bef2-75zgj\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.577317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.577279    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tm2z\" (UniqueName: \"kubernetes.io/projected/7a2df923-20e0-4c9c-bc84-94477db94d59-kube-api-access-2tm2z\") pod \"maas-api-revoke-sub-5240bef2-75zgj\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.577385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.577334    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5240bef2-75zgj\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.663793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.663764    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:26.678999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.678972    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tm2z\" (UniqueName: \"kubernetes.io/projected/7a2df923-20e0-4c9c-bc84-94477db94d59-kube-api-access-2tm2z\") pod \"maas-api-revoke-sub-5240bef2-75zgj\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.679100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.679036    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5240bef2-75zgj\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.679298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.679259    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5240bef2-75zgj\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.679725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.679703    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5240bef2-75zgj\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.681625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.681605    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5240bef2-75zgj\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.690944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.690921    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tm2z\" (UniqueName: \"kubernetes.io/projected/7a2df923-20e0-4c9c-bc84-94477db94d59-kube-api-access-2tm2z\") pod \"maas-api-revoke-sub-5240bef2-75zgj\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.822387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.822323    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:26.844855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.844821    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:26.905254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.904044    5420 generic.go:356] "Generic (PLEG): container finished" podID="e4c7a29d-e4ec-44bc-8e5e-3a349486fa51" containerID="74ffff3291d803106b58f56b32f923e83a7210b4eeab25714ceae665f3ce4d37" exitCode=0
Nov 03 13:27:26.905254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:26.904090    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7" event={"ID":"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51","Type":"ContainerDied","Data":"74ffff3291d803106b58f56b32f923e83a7210b4eeab25714ceae665f3ce4d37"}
Nov 03 13:27:27.013894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:27.013860    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:27.120933 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:27:27.120905    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2df923_20e0_4c9c_bc84_94477db94d59.slice/crio-1254e26ea598ee288375f2e384fe05feb710307fd870f97e667ca10cbe426904 WatchSource:0}: Error finding container 1254e26ea598ee288375f2e384fe05feb710307fd870f97e667ca10cbe426904: Status 404 returned error can't find the container with id 1254e26ea598ee288375f2e384fe05feb710307fd870f97e667ca10cbe426904
Nov 03 13:27:27.121716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:27.121697    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"]
Nov 03 13:27:27.627853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:27.627816    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:27.919041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:27.919006    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj" event={"ID":"7a2df923-20e0-4c9c-bc84-94477db94d59","Type":"ContainerStarted","Data":"88c48efd6a19c1b90bd47d2fb84bee0d59b1f2a6222e325eda708dbbb73c5353"}
Nov 03 13:27:27.919492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:27.919070    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj" event={"ID":"7a2df923-20e0-4c9c-bc84-94477db94d59","Type":"ContainerStarted","Data":"1254e26ea598ee288375f2e384fe05feb710307fd870f97e667ca10cbe426904"}
Nov 03 13:27:27.944411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:27.944358    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj" podStartSLOduration=1.944341198 podStartE2EDuration="1.944341198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:27.939197691 +0000 UTC m=+1089.016962717" watchObservedRunningTime="2026-11-03 13:27:27.944341198 +0000 UTC m=+1089.022106261"
Nov 03 13:27:28.086060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.086039    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:28.203956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.203924    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-kube-api-access-gs57j\" (UniqueName: \"kubernetes.io/projected/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-kube-api-access-gs57j\") pod \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") "
Nov 03 13:27:28.204267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.204093    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-openshift-ca\") pod \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") "
Nov 03 13:27:28.204267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.204154    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-xks-ca\") pod \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\" (UID: \"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51\") "
Nov 03 13:27:28.204608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.204575    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-openshift-ca" pod "e4c7a29d-e4ec-44bc-8e5e-3a349486fa51" (UID: "e4c7a29d-e4ec-44bc-8e5e-3a349486fa51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:28.204868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.204850    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:28.206171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.206150    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-kube-api-access-gs57j" pod "e4c7a29d-e4ec-44bc-8e5e-3a349486fa51" (UID: "e4c7a29d-e4ec-44bc-8e5e-3a349486fa51"). InnerVolumeSpecName "kube-api-access-gs57j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:28.206368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.206348    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-xks-ca" pod "e4c7a29d-e4ec-44bc-8e5e-3a349486fa51" (UID: "e4c7a29d-e4ec-44bc-8e5e-3a349486fa51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:27:28.309797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.309046    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs57j\" (UniqueName: \"kubernetes.io/projected/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-kube-api-access-gs57j\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:28.309797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.309084    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:28.326130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.325333    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:28.929831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.929802    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"
Nov 03 13:27:28.929831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.929820    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7" event={"ID":"e4c7a29d-e4ec-44bc-8e5e-3a349486fa51","Type":"ContainerDied","Data":"d8da2c11c6ec0708747e9238d68441d20b84c1e21c1444a16798436166ddffb2"}
Nov 03 13:27:28.930410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.929869    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8da2c11c6ec0708747e9238d68441d20b84c1e21c1444a16798436166ddffb2"
Nov 03 13:27:28.931809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.931782    5420 generic.go:356] "Generic (PLEG): container finished" podID="7a2df923-20e0-4c9c-bc84-94477db94d59" containerID="88c48efd6a19c1b90bd47d2fb84bee0d59b1f2a6222e325eda708dbbb73c5353" exitCode=0
Nov 03 13:27:28.931898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.931859    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj" event={"ID":"7a2df923-20e0-4c9c-bc84-94477db94d59","Type":"ContainerDied","Data":"88c48efd6a19c1b90bd47d2fb84bee0d59b1f2a6222e325eda708dbbb73c5353"}
Nov 03 13:27:28.981268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:28.974664    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:29.635188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:29.635154    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:30.110482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.110459    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:30.236084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.236046    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-openshift-ca\") pod \"7a2df923-20e0-4c9c-bc84-94477db94d59\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") "
Nov 03 13:27:30.236301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.236099    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2df923-20e0-4c9c-bc84-94477db94d59-kube-api-access-2tm2z\" (UniqueName: \"kubernetes.io/projected/7a2df923-20e0-4c9c-bc84-94477db94d59-kube-api-access-2tm2z\") pod \"7a2df923-20e0-4c9c-bc84-94477db94d59\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") "
Nov 03 13:27:30.236301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.236237    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-xks-ca\") pod \"7a2df923-20e0-4c9c-bc84-94477db94d59\" (UID: \"7a2df923-20e0-4c9c-bc84-94477db94d59\") "
Nov 03 13:27:30.236492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.236467    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-openshift-ca" pod "7a2df923-20e0-4c9c-bc84-94477db94d59" (UID: "7a2df923-20e0-4c9c-bc84-94477db94d59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:30.237228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.237193    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:30.238738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.238712    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-xks-ca" pod "7a2df923-20e0-4c9c-bc84-94477db94d59" (UID: "7a2df923-20e0-4c9c-bc84-94477db94d59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:27:30.239882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.239187    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2df923-20e0-4c9c-bc84-94477db94d59-kube-api-access-2tm2z" pod "7a2df923-20e0-4c9c-bc84-94477db94d59" (UID: "7a2df923-20e0-4c9c-bc84-94477db94d59"). InnerVolumeSpecName "kube-api-access-2tm2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:30.242000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.241975    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:30.338566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.338538    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tm2z\" (UniqueName: \"kubernetes.io/projected/7a2df923-20e0-4c9c-bc84-94477db94d59-kube-api-access-2tm2z\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:30.338566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.338561    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2df923-20e0-4c9c-bc84-94477db94d59-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:30.860994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.860961    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:30.951336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.951307    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"
Nov 03 13:27:30.951532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.951338    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj" event={"ID":"7a2df923-20e0-4c9c-bc84-94477db94d59","Type":"ContainerDied","Data":"1254e26ea598ee288375f2e384fe05feb710307fd870f97e667ca10cbe426904"}
Nov 03 13:27:30.951532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:30.951395    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1254e26ea598ee288375f2e384fe05feb710307fd870f97e667ca10cbe426904"
Nov 03 13:27:31.537083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:31.537031    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:32.215261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:32.215230    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:32.834467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:32.834430    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:33.106890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:33.106813    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"]
Nov 03 13:27:33.119133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:33.119106    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f064-vrw8l"]
Nov 03 13:27:33.661380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:33.661347    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:34.034706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:34.034658    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:34.222873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:34.222843    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:34.485887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:34.485857    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:34.648233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:34.647923    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:35.005737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:35.005707    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:35.025242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:35.025153    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"]
Nov 03 13:27:35.033931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:35.033901    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320dd917-p678n"]
Nov 03 13:27:35.085833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:35.085804    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ee43d5-98d0-4510-8558-848e406f9f90" path="/var/lib/kubelet/pods/31ee43d5-98d0-4510-8558-848e406f9f90/volumes"
Nov 03 13:27:35.086431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:35.086414    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49920094-748c-490a-a7dc-ded3101add86" path="/var/lib/kubelet/pods/49920094-748c-490a-a7dc-ded3101add86/volumes"
Nov 03 13:27:35.295939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:35.295863    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:35.434416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:35.434385    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:35.757884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:35.757854    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:36.405404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.405371    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:36.482630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.482601    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"]
Nov 03 13:27:36.483371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.483349    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e4c7a29d-e4ec-44bc-8e5e-3a349486fa51" containerName="revoke-keys"
Nov 03 13:27:36.483371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.483370    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c7a29d-e4ec-44bc-8e5e-3a349486fa51" containerName="revoke-keys"
Nov 03 13:27:36.483582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.483404    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7a2df923-20e0-4c9c-bc84-94477db94d59" containerName="revoke-keys"
Nov 03 13:27:36.483582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.483412    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2df923-20e0-4c9c-bc84-94477db94d59" containerName="revoke-keys"
Nov 03 13:27:36.483690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.483635    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c7a29d-e4ec-44bc-8e5e-3a349486fa51" containerName="revoke-keys"
Nov 03 13:27:36.483690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.483658    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2df923-20e0-4c9c-bc84-94477db94d59" containerName="revoke-keys"
Nov 03 13:27:36.487568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.487549    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.514259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.514197    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"]
Nov 03 13:27:36.543159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.543093    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.543574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.543196    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.543574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.543259    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vn2x\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-kube-api-access-8vn2x\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.543574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.543290    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.543574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.543346    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.543574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.543415    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.543574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.543488    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829b972d-478f-49e8-a7eb-678c3db13df8-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.543574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.543516    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829b972d-478f-49e8-a7eb-678c3db13df8-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.543574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.543535    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.644709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.644672    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.644881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.644757    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.644881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.644843    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829b972d-478f-49e8-a7eb-678c3db13df8-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.644881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.644869    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829b972d-478f-49e8-a7eb-678c3db13df8-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.644887    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.645037    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.645081    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.645134    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vn2x\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-kube-api-access-8vn2x\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.645165    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.645186    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.645262    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.645454    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.645610    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.645707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.645675    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829b972d-478f-49e8-a7eb-678c3db13df8-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.647332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.647311    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.647463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.647409    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829b972d-478f-49e8-a7eb-678c3db13df8-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.654366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.654340    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.656783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.656729    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vn2x\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-kube-api-access-8vn2x\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.809437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.809407    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:36.906108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:36.905624    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:27:37.122857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:37.122826    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"]
Nov 03 13:27:37.125869 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:27:37.125834    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829b972d_478f_49e8_a7eb_678c3db13df8.slice/crio-c8a5af70a3c2984b3c81f3660fe36eda266650885005037d98e0911083e905b0 WatchSource:0}: Error finding container c8a5af70a3c2984b3c81f3660fe36eda266650885005037d98e0911083e905b0: Status 404 returned error can't find the container with id c8a5af70a3c2984b3c81f3660fe36eda266650885005037d98e0911083e905b0
Nov 03 13:27:37.134175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:37.134029    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:27:37.134399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:37.134371    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:27:37.134496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:37.134474    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:27:37.249806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:37.249772    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:38.077728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.075987    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"]
Nov 03 13:27:38.090140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.088424    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"]
Nov 03 13:27:38.090140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.088530    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.094670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.094646    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:27:38.094929 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.094912    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:27:38.095121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.095107    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:27:38.095373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.095356    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:27:38.115458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.115422    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" event={"ID":"829b972d-478f-49e8-a7eb-678c3db13df8","Type":"ContainerStarted","Data":"4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea"}
Nov 03 13:27:38.116087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.116064    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" event={"ID":"829b972d-478f-49e8-a7eb-678c3db13df8","Type":"ContainerStarted","Data":"c8a5af70a3c2984b3c81f3660fe36eda266650885005037d98e0911083e905b0"}
Nov 03 13:27:38.151235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.148690    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" podStartSLOduration=2.148669083 podStartE2EDuration="2.148669083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:38.148509075 +0000 UTC m=+1099.226274103" watchObservedRunningTime="2026-11-03 13:27:38.148669083 +0000 UTC m=+1099.226434114"
Nov 03 13:27:38.215328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.215243    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bb5bbd-vqwhl\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.217025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.217000    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bb5bbd-vqwhl\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.217524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.217506    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkhv\" (UniqueName: \"kubernetes.io/projected/3a899091-6229-47e5-b778-7a668b28e4d9-kube-api-access-pmkhv\") pod \"maas-api-revoke-sub-83bb5bbd-vqwhl\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.338277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.321186    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bb5bbd-vqwhl\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.338277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.321363    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bb5bbd-vqwhl\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.338277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.321449    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkhv\" (UniqueName: \"kubernetes.io/projected/3a899091-6229-47e5-b778-7a668b28e4d9-kube-api-access-pmkhv\") pod \"maas-api-revoke-sub-83bb5bbd-vqwhl\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.338277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.322854    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bb5bbd-vqwhl\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.357836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.353108    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkhv\" (UniqueName: \"kubernetes.io/projected/3a899091-6229-47e5-b778-7a668b28e4d9-kube-api-access-pmkhv\") pod \"maas-api-revoke-sub-83bb5bbd-vqwhl\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.357836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.353556    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bb5bbd-vqwhl\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.418537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.418478    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:38.776252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.775682    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:38.815646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.815094    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 13:27:38.815646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.815154    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 13:27:38.815646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.815262    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:38.943257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:38.942686    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:39.547737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:39.547600    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:39.579889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:39.579857    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"]
Nov 03 13:27:39.580156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:39.580138    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89922625-9qrgl"]
Nov 03 13:27:39.716888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:39.716854    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"]
Nov 03 13:27:39.811177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:39.811077    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 13:27:39.811177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:39.811131    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 13:27:40.410236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:40.403096    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:40.756236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:40.749833    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:40.810973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:40.810440    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 13:27:40.810973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:40.810508    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 13:27:40.880378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:40.837976    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl" event={"ID":"3a899091-6229-47e5-b778-7a668b28e4d9","Type":"ContainerStarted","Data":"cd3fc9b8327e0e4cd02e23a45b214610f904c91630eb0d8b2f39c4b9839da558"}
Nov 03 13:27:40.880378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:40.838067    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl" event={"ID":"3a899091-6229-47e5-b778-7a668b28e4d9","Type":"ContainerStarted","Data":"80d4e0cd9eacedcd50255bf5b3bc5afb2892e73b253dbb23b6c3a6d75fae386a"}
Nov 03 13:27:40.914371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:40.914321    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl" podStartSLOduration=2.914303227 podStartE2EDuration="2.914303227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:40.904627298 +0000 UTC m=+1101.982392325" watchObservedRunningTime="2026-11-03 13:27:40.914303227 +0000 UTC m=+1101.992068252"
Nov 03 13:27:40.994382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:40.994166    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:41.165063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:41.164329    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:41.233766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:41.233729    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9ab0f3-3e08-4d06-b4ac-ffc86671db84" path="/var/lib/kubelet/pods/3d9ab0f3-3e08-4d06-b4ac-ffc86671db84/volumes"
Nov 03 13:27:41.295620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:41.295583    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"]
Nov 03 13:27:41.298577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:41.298550    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c87701ad-ftls7"]
Nov 03 13:27:41.328927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:41.328665    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:41.810584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:41.810547    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 13:27:41.811201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:41.811166    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 13:27:41.869321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:41.869285    5420 generic.go:356] "Generic (PLEG): container finished" podID="3a899091-6229-47e5-b778-7a668b28e4d9" containerID="cd3fc9b8327e0e4cd02e23a45b214610f904c91630eb0d8b2f39c4b9839da558" exitCode=0
Nov 03 13:27:41.869558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:41.869494    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl" event={"ID":"3a899091-6229-47e5-b778-7a668b28e4d9","Type":"ContainerDied","Data":"cd3fc9b8327e0e4cd02e23a45b214610f904c91630eb0d8b2f39c4b9839da558"}
Nov 03 13:27:42.786879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:42.786835    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:42.810301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:42.810193    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 13:27:42.810461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:42.810308    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 13:27:42.981916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:42.981885    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:43.091696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.091664    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c7a29d-e4ec-44bc-8e5e-3a349486fa51" path="/var/lib/kubelet/pods/e4c7a29d-e4ec-44bc-8e5e-3a349486fa51/volumes"
Nov 03 13:27:43.104201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.104174    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:43.183170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.183127    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:43.204966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.204932    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-openshift-ca\") pod \"3a899091-6229-47e5-b778-7a668b28e4d9\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") "
Nov 03 13:27:43.205167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.204992    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a899091-6229-47e5-b778-7a668b28e4d9-kube-api-access-pmkhv\" (UniqueName: \"kubernetes.io/projected/3a899091-6229-47e5-b778-7a668b28e4d9-kube-api-access-pmkhv\") pod \"3a899091-6229-47e5-b778-7a668b28e4d9\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") "
Nov 03 13:27:43.205167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.205094    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-xks-ca\") pod \"3a899091-6229-47e5-b778-7a668b28e4d9\" (UID: \"3a899091-6229-47e5-b778-7a668b28e4d9\") "
Nov 03 13:27:43.210152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.208665    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-openshift-ca" pod "3a899091-6229-47e5-b778-7a668b28e4d9" (UID: "3a899091-6229-47e5-b778-7a668b28e4d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:43.309426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.309381    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a899091-6229-47e5-b778-7a668b28e4d9-kube-api-access-pmkhv" pod "3a899091-6229-47e5-b778-7a668b28e4d9" (UID: "3a899091-6229-47e5-b778-7a668b28e4d9"). InnerVolumeSpecName "kube-api-access-pmkhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:43.378743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.370884    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:43.378743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.370924    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkhv\" (UniqueName: \"kubernetes.io/projected/3a899091-6229-47e5-b778-7a668b28e4d9-kube-api-access-pmkhv\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:43.378743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.376366    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-xks-ca" pod "3a899091-6229-47e5-b778-7a668b28e4d9" (UID: "3a899091-6229-47e5-b778-7a668b28e4d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:27:43.384339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.379821    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:43.480183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.480017    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a899091-6229-47e5-b778-7a668b28e4d9-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:43.810202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.810151    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 13:27:43.810413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.810296    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 13:27:43.889108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.889084    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"
Nov 03 13:27:43.889289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.889110    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl" event={"ID":"3a899091-6229-47e5-b778-7a668b28e4d9","Type":"ContainerDied","Data":"80d4e0cd9eacedcd50255bf5b3bc5afb2892e73b253dbb23b6c3a6d75fae386a"}
Nov 03 13:27:43.889289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:43.889164    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d4e0cd9eacedcd50255bf5b3bc5afb2892e73b253dbb23b6c3a6d75fae386a"
Nov 03 13:27:44.069619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:44.069541    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:44.681491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:44.681459    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:45.046161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:45.046135    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:45.046708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:45.046684    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:45.047515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:45.047496    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:27:45.672950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:45.672917    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:46.460459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:46.460426    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:47.045200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:47.045163    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"]
Nov 03 13:27:47.053164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:47.053133    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce326e7b-64sbd"]
Nov 03 13:27:47.085156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:47.085128    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6" path="/var/lib/kubelet/pods/f3d7ce81-44e4-4b60-82bb-87b4b8e0cac6/volumes"
Nov 03 13:27:47.266747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:47.266714    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:47.997715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:47.997687    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:48.720775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.720747    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:48.824764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.824729    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"]
Nov 03 13:27:48.825645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.825626    5420 cpu_manager.go:414] "containerMap: removing container" podUID="3a899091-6229-47e5-b778-7a668b28e4d9" containerName="revoke-keys"
Nov 03 13:27:48.825645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.825645    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a899091-6229-47e5-b778-7a668b28e4d9" containerName="revoke-keys"
Nov 03 13:27:48.825862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.825781    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a899091-6229-47e5-b778-7a668b28e4d9" containerName="revoke-keys"
Nov 03 13:27:48.831073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.831051    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:48.835097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.835073    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"]
Nov 03 13:27:48.837915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.837856    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:27:48.837915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.837872    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:27:48.838072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.837919    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:27:48.838072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.837969    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:27:48.883612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.883578    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ce8aa5-6fjkx\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:48.883821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.883798    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpnw\" (UniqueName: \"kubernetes.io/projected/23cb02a4-b31f-45c6-8763-5b479b788a3f-kube-api-access-8jpnw\") pod \"maas-api-revoke-sub-65ce8aa5-6fjkx\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:48.883980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.883960    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ce8aa5-6fjkx\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:48.985452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.985365    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ce8aa5-6fjkx\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:48.985601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.985499    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ce8aa5-6fjkx\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:48.985643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.985596    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpnw\" (UniqueName: \"kubernetes.io/projected/23cb02a4-b31f-45c6-8763-5b479b788a3f-kube-api-access-8jpnw\") pod \"maas-api-revoke-sub-65ce8aa5-6fjkx\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:48.986108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.986079    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ce8aa5-6fjkx\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:48.987846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.987824    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ce8aa5-6fjkx\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:48.994399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:48.994375    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpnw\" (UniqueName: \"kubernetes.io/projected/23cb02a4-b31f-45c6-8763-5b479b788a3f-kube-api-access-8jpnw\") pod \"maas-api-revoke-sub-65ce8aa5-6fjkx\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:49.159470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:49.159436    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:49.444939 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:27:49.444908    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cb02a4_b31f_45c6_8763_5b479b788a3f.slice/crio-70d5d7269ba0da5a42d48f50cfecb75b98a0b77381074750611a9b11dc629dee WatchSource:0}: Error finding container 70d5d7269ba0da5a42d48f50cfecb75b98a0b77381074750611a9b11dc629dee: Status 404 returned error can't find the container with id 70d5d7269ba0da5a42d48f50cfecb75b98a0b77381074750611a9b11dc629dee
Nov 03 13:27:49.447813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:49.447787    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"]
Nov 03 13:27:49.626271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:49.626239    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:49.948920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:49.948885    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx" event={"ID":"23cb02a4-b31f-45c6-8763-5b479b788a3f","Type":"ContainerStarted","Data":"ffd480ebb0d618888cadc9b7ebacc43672976bd01b5accad0cdd20d325efc24b"}
Nov 03 13:27:49.949326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:49.948936    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx" event={"ID":"23cb02a4-b31f-45c6-8763-5b479b788a3f","Type":"ContainerStarted","Data":"70d5d7269ba0da5a42d48f50cfecb75b98a0b77381074750611a9b11dc629dee"}
Nov 03 13:27:49.971974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:49.971915    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx" podStartSLOduration=1.9719008 podStartE2EDuration="1.9719008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:49.970726787 +0000 UTC m=+1111.048491814" watchObservedRunningTime="2026-11-03 13:27:49.9719008 +0000 UTC m=+1111.049665826"
Nov 03 13:27:50.425966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:50.425938    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:50.959355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:50.959321    5420 generic.go:356] "Generic (PLEG): container finished" podID="23cb02a4-b31f-45c6-8763-5b479b788a3f" containerID="ffd480ebb0d618888cadc9b7ebacc43672976bd01b5accad0cdd20d325efc24b" exitCode=0
Nov 03 13:27:50.959771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:50.959426    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx" event={"ID":"23cb02a4-b31f-45c6-8763-5b479b788a3f","Type":"ContainerDied","Data":"ffd480ebb0d618888cadc9b7ebacc43672976bd01b5accad0cdd20d325efc24b"}
Nov 03 13:27:51.059040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:51.059010    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"]
Nov 03 13:27:51.070896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:51.070845    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387e50b1-t497j"]
Nov 03 13:27:51.089043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:51.089013    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c06aeab-f308-4003-adbb-3df04b94c66b" path="/var/lib/kubelet/pods/4c06aeab-f308-4003-adbb-3df04b94c66b/volumes"
Nov 03 13:27:51.189069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:51.189031    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:51.369915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:51.369828    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:51.528981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:51.528934    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:52.166169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.166147    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:52.234741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.234709    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cb02a4-b31f-45c6-8763-5b479b788a3f-kube-api-access-8jpnw\" (UniqueName: \"kubernetes.io/projected/23cb02a4-b31f-45c6-8763-5b479b788a3f-kube-api-access-8jpnw\") pod \"23cb02a4-b31f-45c6-8763-5b479b788a3f\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") "
Nov 03 13:27:52.234978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.234833    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-xks-ca\") pod \"23cb02a4-b31f-45c6-8763-5b479b788a3f\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") "
Nov 03 13:27:52.234978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.234929    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-openshift-ca\") pod \"23cb02a4-b31f-45c6-8763-5b479b788a3f\" (UID: \"23cb02a4-b31f-45c6-8763-5b479b788a3f\") "
Nov 03 13:27:52.235592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.235556    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-openshift-ca" pod "23cb02a4-b31f-45c6-8763-5b479b788a3f" (UID: "23cb02a4-b31f-45c6-8763-5b479b788a3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:27:52.235829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.235802    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:52.237609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.237577    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cb02a4-b31f-45c6-8763-5b479b788a3f-kube-api-access-8jpnw" pod "23cb02a4-b31f-45c6-8763-5b479b788a3f" (UID: "23cb02a4-b31f-45c6-8763-5b479b788a3f"). InnerVolumeSpecName "kube-api-access-8jpnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:27:52.237721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.237648    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-xks-ca" pod "23cb02a4-b31f-45c6-8763-5b479b788a3f" (UID: "23cb02a4-b31f-45c6-8763-5b479b788a3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:27:52.245339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.245262    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:52.337667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.337627    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jpnw\" (UniqueName: \"kubernetes.io/projected/23cb02a4-b31f-45c6-8763-5b479b788a3f-kube-api-access-8jpnw\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:52.337667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.337666    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb02a4-b31f-45c6-8763-5b479b788a3f-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:27:52.998273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.998247    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"
Nov 03 13:27:52.998436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.998244    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx" event={"ID":"23cb02a4-b31f-45c6-8763-5b479b788a3f","Type":"ContainerDied","Data":"70d5d7269ba0da5a42d48f50cfecb75b98a0b77381074750611a9b11dc629dee"}
Nov 03 13:27:52.998436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:52.998388    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d5d7269ba0da5a42d48f50cfecb75b98a0b77381074750611a9b11dc629dee"
Nov 03 13:27:53.103359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:53.103331    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:53.444256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:53.444163    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:54.143725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:54.143694    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:54.733514 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:27:54.733484    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:27:54.838117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:54.838083    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:55.520257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:55.520132    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:56.224036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:56.224007    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:56.426863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:56.426822    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:56.716584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:56.716550    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:57.630907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:57.630869    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:58.084646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.084614    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"]
Nov 03 13:27:58.106193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.106138    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-494fd49f-dvwf5"]
Nov 03 13:27:58.110845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.110819    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"]
Nov 03 13:27:58.112093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.112068    5420 cpu_manager.go:414] "containerMap: removing container" podUID="23cb02a4-b31f-45c6-8763-5b479b788a3f" containerName="revoke-keys"
Nov 03 13:27:58.112258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.112244    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cb02a4-b31f-45c6-8763-5b479b788a3f" containerName="revoke-keys"
Nov 03 13:27:58.112598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.112582    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cb02a4-b31f-45c6-8763-5b479b788a3f" containerName="revoke-keys"
Nov 03 13:27:58.132494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.132470    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.142339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.142010    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"openshift-service-ca.crt\""
Nov 03 13:27:58.142339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.142130    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1a18c8\""
Nov 03 13:27:58.142339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.142187    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"kube-root-ca.crt\""
Nov 03 13:27:58.142339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.142273    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"routing-overlay\""
Nov 03 13:27:58.142932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.142787    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-e2e--d832c86e-dockercfg-sq45c\""
Nov 03 13:27:58.142932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.142854    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-31f9f21f7966\""
Nov 03 13:27:58.142932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.142903    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-de83083e156c\""
Nov 03 13:27:58.154491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.154465    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"]
Nov 03 13:27:58.277236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.276345    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjxb\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-kube-api-access-rjjxb\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.277236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.276408    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.277236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.276653    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.277236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.276800    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.353308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.353185    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:58.378363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.378323    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.378579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.378543    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjxb\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-kube-api-access-rjjxb\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.378669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.378586    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.378832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.378812    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.379540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.379508    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.380054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.380029    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.389819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.382277    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.397322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.397282    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjxb\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-kube-api-access-rjjxb\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.491768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.491728    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:58.819556 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:27:58.819520    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee30c13_51fe_46ef_a7ec_eebbd05f4644.slice/crio-ea3626d038248ee375a3c970348a71dce15ce51523200c44bd25b18c137dba51 WatchSource:0}: Error finding container ea3626d038248ee375a3c970348a71dce15ce51523200c44bd25b18c137dba51: Status 404 returned error can't find the container with id ea3626d038248ee375a3c970348a71dce15ce51523200c44bd25b18c137dba51
Nov 03 13:27:58.820369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:58.820342    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"]
Nov 03 13:27:59.096492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:59.096407    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6e71b5-64a3-4276-8380-328c42b81016" path="/var/lib/kubelet/pods/6c6e71b5-64a3-4276-8380-328c42b81016/volumes"
Nov 03 13:27:59.097802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:59.097774    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg" event={"ID":"bee30c13-51fe-46ef-a7ec-eebbd05f4644","Type":"ContainerStarted","Data":"c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1"}
Nov 03 13:27:59.097947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:59.097816    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg" event={"ID":"bee30c13-51fe-46ef-a7ec-eebbd05f4644","Type":"ContainerStarted","Data":"ea3626d038248ee375a3c970348a71dce15ce51523200c44bd25b18c137dba51"}
Nov 03 13:27:59.098037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:59.098019    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:27:59.119227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:59.119174    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg" podStartSLOduration=1.11915803 podStartE2EDuration="1.11915803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:59.118126661 +0000 UTC m=+1120.195891698" watchObservedRunningTime="2026-11-03 13:27:59.11915803 +0000 UTC m=+1120.196923055"
Nov 03 13:27:59.687739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:27:59.687702    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:00.324137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:00.324103    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:01.110390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:01.110319    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:01.305993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:01.305956    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:01.800802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:01.800774    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:01.960572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:01.960518    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:02.206294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:02.206262    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:02.933860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:02.933822    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:03.297106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.297067    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"]
Nov 03 13:28:03.303105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.303078    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.320416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.320391    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:28:03.321395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.321330    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:28:03.322396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.322375    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:28:03.322669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.322652    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:28:03.323152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.323133    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"]
Nov 03 13:28:03.366845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.366815    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539e8ee4-7b84m\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.367000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.366928    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mscj\" (UniqueName: \"kubernetes.io/projected/1885eb48-0882-4f66-ba99-27bae5c8a95f-kube-api-access-4mscj\") pod \"maas-api-revoke-sub-539e8ee4-7b84m\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.367078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.367053    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539e8ee4-7b84m\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.468485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.468457    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539e8ee4-7b84m\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.468604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.468506    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mscj\" (UniqueName: \"kubernetes.io/projected/1885eb48-0882-4f66-ba99-27bae5c8a95f-kube-api-access-4mscj\") pod \"maas-api-revoke-sub-539e8ee4-7b84m\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.468604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.468556    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539e8ee4-7b84m\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.469080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.469053    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539e8ee4-7b84m\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.470752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.470730    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539e8ee4-7b84m\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.476896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.476872    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mscj\" (UniqueName: \"kubernetes.io/projected/1885eb48-0882-4f66-ba99-27bae5c8a95f-kube-api-access-4mscj\") pod \"maas-api-revoke-sub-539e8ee4-7b84m\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:03.630413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:03.630336    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:04.022921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:04.022878    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:04.066630 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:28:04.066595    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1885eb48_0882_4f66_ba99_27bae5c8a95f.slice/crio-bc503b890bc6a525f2ccc03724314d4f4bc074dc39e3a13bcc288ae420c9e0ca WatchSource:0}: Error finding container bc503b890bc6a525f2ccc03724314d4f4bc074dc39e3a13bcc288ae420c9e0ca: Status 404 returned error can't find the container with id bc503b890bc6a525f2ccc03724314d4f4bc074dc39e3a13bcc288ae420c9e0ca
Nov 03 13:28:04.067782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:04.067758    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"]
Nov 03 13:28:04.151674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:04.151638    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m" event={"ID":"1885eb48-0882-4f66-ba99-27bae5c8a95f","Type":"ContainerStarted","Data":"bc503b890bc6a525f2ccc03724314d4f4bc074dc39e3a13bcc288ae420c9e0ca"}
Nov 03 13:28:04.747342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:04.747309    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:05.163349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:05.163318    5420 generic.go:356] "Generic (PLEG): container finished" podID="1885eb48-0882-4f66-ba99-27bae5c8a95f" containerID="8310b59a1de1c8eb922ce5e3f04e882d173331f3a46eaf3281e3b0c108c82d13" exitCode=0
Nov 03 13:28:05.163766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:05.163398    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m" event={"ID":"1885eb48-0882-4f66-ba99-27bae5c8a95f","Type":"ContainerDied","Data":"8310b59a1de1c8eb922ce5e3f04e882d173331f3a46eaf3281e3b0c108c82d13"}
Nov 03 13:28:05.628825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:05.628790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:06.350825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.350796    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:06.429009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.428918    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:06.448613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.448564    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-xks-ca\") pod \"1885eb48-0882-4f66-ba99-27bae5c8a95f\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") "
Nov 03 13:28:06.448826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.448682    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1885eb48-0882-4f66-ba99-27bae5c8a95f-kube-api-access-4mscj\" (UniqueName: \"kubernetes.io/projected/1885eb48-0882-4f66-ba99-27bae5c8a95f-kube-api-access-4mscj\") pod \"1885eb48-0882-4f66-ba99-27bae5c8a95f\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") "
Nov 03 13:28:06.448907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.448872    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-openshift-ca\") pod \"1885eb48-0882-4f66-ba99-27bae5c8a95f\" (UID: \"1885eb48-0882-4f66-ba99-27bae5c8a95f\") "
Nov 03 13:28:06.450059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.450028    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-openshift-ca" pod "1885eb48-0882-4f66-ba99-27bae5c8a95f" (UID: "1885eb48-0882-4f66-ba99-27bae5c8a95f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:06.452063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.452036    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1885eb48-0882-4f66-ba99-27bae5c8a95f-kube-api-access-4mscj" pod "1885eb48-0882-4f66-ba99-27bae5c8a95f" (UID: "1885eb48-0882-4f66-ba99-27bae5c8a95f"). InnerVolumeSpecName "kube-api-access-4mscj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:06.453140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.453119    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-xks-ca" pod "1885eb48-0882-4f66-ba99-27bae5c8a95f" (UID: "1885eb48-0882-4f66-ba99-27bae5c8a95f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:28:06.551672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.551637    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:06.551672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.551669    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1885eb48-0882-4f66-ba99-27bae5c8a95f-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:06.551672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:06.551679    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mscj\" (UniqueName: \"kubernetes.io/projected/1885eb48-0882-4f66-ba99-27bae5c8a95f-kube-api-access-4mscj\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:07.105481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:07.105430    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:07.184613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:07.184585    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"
Nov 03 13:28:07.184792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:07.184624    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m" event={"ID":"1885eb48-0882-4f66-ba99-27bae5c8a95f","Type":"ContainerDied","Data":"bc503b890bc6a525f2ccc03724314d4f4bc074dc39e3a13bcc288ae420c9e0ca"}
Nov 03 13:28:07.184792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:07.184669    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc503b890bc6a525f2ccc03724314d4f4bc074dc39e3a13bcc288ae420c9e0ca"
Nov 03 13:28:07.562429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:07.562400    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:07.791249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:07.790173    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:08.640587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:08.635386    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:09.084636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:09.084606    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:09.377175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:09.377085    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:10.049073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.049042    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"]
Nov 03 13:28:10.054010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.053973    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef380069-zzqvf"]
Nov 03 13:28:10.110858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.110832    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:28:10.181117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.181087    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:10.480806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.480772    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:10.874782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.874690    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:10.896508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.896478    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"]
Nov 03 13:28:10.897173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.897161    5420 cpu_manager.go:414] "containerMap: removing container" podUID="1885eb48-0882-4f66-ba99-27bae5c8a95f" containerName="revoke-keys"
Nov 03 13:28:10.897271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.897174    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1885eb48-0882-4f66-ba99-27bae5c8a95f" containerName="revoke-keys"
Nov 03 13:28:10.897366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.897356    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1885eb48-0882-4f66-ba99-27bae5c8a95f" containerName="revoke-keys"
Nov 03 13:28:10.902550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.902533    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:10.916561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.916540    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-cf53a4-openshift-default-dockercfg-8z7h7\""
Nov 03 13:28:10.929765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:10.929740    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"]
Nov 03 13:28:11.017250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.016330    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istiod-ca-cert\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.017250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.016901    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-token\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.017250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.016995    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-socket\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.017250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.017031    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-credential-socket\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.017250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.017073    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-podinfo\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.017250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.017145    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-data\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.017250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.017175    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-certs\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.017250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.017258    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-envoy\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.017757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.017320    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tf2k\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-kube-api-access-4tf2k\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.126283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.119130    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-credential-socket\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.126283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.119201    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-credential-socket\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.126283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.119291    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-podinfo\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.136341    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-data\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.136397    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-certs\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.136461    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-envoy\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.136532    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tf2k\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-kube-api-access-4tf2k\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.136577    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istiod-ca-cert\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.136826    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-token\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.136925    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-socket\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.137350    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-socket\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.138047    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istiod-ca-cert\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.138820    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-certs\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.139358    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-data\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.144771    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-envoy\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.148241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.145268    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-podinfo\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.159453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.151302    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:11.159453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.153155    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-token\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.159453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.157473    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tf2k\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-kube-api-access-4tf2k\") pod \"e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") " pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.159453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.157589    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0508373e-4df4-462d-815a-e2dccd53388e" path="/var/lib/kubelet/pods/0508373e-4df4-462d-815a-e2dccd53388e/volumes"
Nov 03 13:28:11.219342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.218813    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:11.320079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.320011    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:11.528664 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:28:11.528633    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b26dd89_8856_4b20_b5be_62a4abf9c9a6.slice/crio-c1c5e758368f53a62ec0b10b22a47da02697794682ec93938d74b299431bfeb2 WatchSource:0}: Error finding container c1c5e758368f53a62ec0b10b22a47da02697794682ec93938d74b299431bfeb2: Status 404 returned error can't find the container with id c1c5e758368f53a62ec0b10b22a47da02697794682ec93938d74b299431bfeb2
Nov 03 13:28:11.529750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.529719    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"]
Nov 03 13:28:11.531446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.531411    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:28:11.531545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.531495    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:28:11.531545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.531537    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:28:11.707528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:11.707501    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:12.238668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:12.238635    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" event={"ID":"0b26dd89-8856-4b20-b5be-62a4abf9c9a6","Type":"ContainerStarted","Data":"90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98"}
Nov 03 13:28:12.239032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:12.238685    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" event={"ID":"0b26dd89-8856-4b20-b5be-62a4abf9c9a6","Type":"ContainerStarted","Data":"c1c5e758368f53a62ec0b10b22a47da02697794682ec93938d74b299431bfeb2"}
Nov 03 13:28:12.273284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:12.272994    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" podStartSLOduration=2.272975155 podStartE2EDuration="2.272975155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:12.265348969 +0000 UTC m=+1133.343114007" watchObservedRunningTime="2026-11-03 13:28:12.272975155 +0000 UTC m=+1133.350740184"
Nov 03 13:28:12.280244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:12.280202    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"]
Nov 03 13:28:12.616291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:12.613807    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:13.027547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:13.027515    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"]
Nov 03 13:28:13.032267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:13.032243    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-092ca36a-f999l"]
Nov 03 13:28:13.090032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:13.090001    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195f6c59-4d56-4060-b25e-9c6993cf8977" path="/var/lib/kubelet/pods/195f6c59-4d56-4060-b25e-9c6993cf8977/volumes"
Nov 03 13:28:13.219226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:13.219193    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:13.225589 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:13.225550    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:13.247605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:13.247576    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:13.248453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:13.248434    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:28:13.364137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:13.364043    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:14.163184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:14.163153    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:14.559393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:14.559365    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:14.851540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:14.851453    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:15.204658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.204625    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"]
Nov 03 13:28:15.210850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.210817    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.218274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.216145    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:28:15.218274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.217196    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:28:15.219364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.219347    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:28:15.219493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.219389    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:28:15.222422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.222400    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"]
Nov 03 13:28:15.326973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.326946    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa4505f-9jnln\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.327256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.327206    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xzdw\" (UniqueName: \"kubernetes.io/projected/2543bc1f-d0ea-4a00-9958-4b20958915b2-kube-api-access-9xzdw\") pod \"maas-api-revoke-sub-afa4505f-9jnln\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.327410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.327361    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa4505f-9jnln\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.429481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.429444    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa4505f-9jnln\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.429630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.429578    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xzdw\" (UniqueName: \"kubernetes.io/projected/2543bc1f-d0ea-4a00-9958-4b20958915b2-kube-api-access-9xzdw\") pod \"maas-api-revoke-sub-afa4505f-9jnln\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.429667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.429633    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa4505f-9jnln\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.430350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.430326    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa4505f-9jnln\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.431841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.431814    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa4505f-9jnln\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.440424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.440403    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xzdw\" (UniqueName: \"kubernetes.io/projected/2543bc1f-d0ea-4a00-9958-4b20958915b2-kube-api-access-9xzdw\") pod \"maas-api-revoke-sub-afa4505f-9jnln\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.533712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.533629    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:15.685255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.684989    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:15.844853 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:28:15.844818    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2543bc1f_d0ea_4a00_9958_4b20958915b2.slice/crio-2f3281fed7c9a7cc76f03d15e4ff1bdb473f3cd57e67f71a7c912219dff884ae WatchSource:0}: Error finding container 2f3281fed7c9a7cc76f03d15e4ff1bdb473f3cd57e67f71a7c912219dff884ae: Status 404 returned error can't find the container with id 2f3281fed7c9a7cc76f03d15e4ff1bdb473f3cd57e67f71a7c912219dff884ae
Nov 03 13:28:15.846816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:15.846711    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"]
Nov 03 13:28:16.290079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.289927    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln" event={"ID":"2543bc1f-d0ea-4a00-9958-4b20958915b2","Type":"ContainerStarted","Data":"a657a449c789666c432852f5b741167d9af09d4c18c6a708ea81fbb33199108c"}
Nov 03 13:28:16.290079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.290042    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln" event={"ID":"2543bc1f-d0ea-4a00-9958-4b20958915b2","Type":"ContainerStarted","Data":"2f3281fed7c9a7cc76f03d15e4ff1bdb473f3cd57e67f71a7c912219dff884ae"}
Nov 03 13:28:16.327266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.325922    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln" podStartSLOduration=1.325905489 podStartE2EDuration="1.325905489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:16.322736089 +0000 UTC m=+1137.400501114" watchObservedRunningTime="2026-11-03 13:28:16.325905489 +0000 UTC m=+1137.403670515"
Nov 03 13:28:16.364458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.364409    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:16.393156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.393089    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"]
Nov 03 13:28:16.397901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.397877    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.419253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.418466    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"]
Nov 03 13:28:16.449428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.449376    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef15d972-vmndv\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.449428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.449422    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jbd\" (UniqueName: \"kubernetes.io/projected/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-kube-api-access-v2jbd\") pod \"maas-api-revoke-keys-e2e-derive-ef15d972-vmndv\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.449708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.449470    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef15d972-vmndv\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.572240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.571620    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef15d972-vmndv\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.572240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.571668    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jbd\" (UniqueName: \"kubernetes.io/projected/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-kube-api-access-v2jbd\") pod \"maas-api-revoke-keys-e2e-derive-ef15d972-vmndv\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.572240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.571708    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef15d972-vmndv\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.588231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.574200    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef15d972-vmndv\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.588231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.583415    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:16.599698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.593046    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef15d972-vmndv\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.636237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.627561    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jbd\" (UniqueName: \"kubernetes.io/projected/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-kube-api-access-v2jbd\") pod \"maas-api-revoke-keys-e2e-derive-ef15d972-vmndv\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.722005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.721976    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:16.865812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:16.860134    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:17.081753 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:28:17.081726    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6d5989_6ed4_448a_a3bd_fd53bf29c3eb.slice/crio-3f7b1da3cabc8cc2748076701b5ca89d165bdf8999f51fbb01a1e1e02075ff78 WatchSource:0}: Error finding container 3f7b1da3cabc8cc2748076701b5ca89d165bdf8999f51fbb01a1e1e02075ff78: Status 404 returned error can't find the container with id 3f7b1da3cabc8cc2748076701b5ca89d165bdf8999f51fbb01a1e1e02075ff78
Nov 03 13:28:17.088180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.088154    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"]
Nov 03 13:28:17.280920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.280882    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh" podUID="0de9561a-b039-49f1-a901-d338e7e49820" containerName="payload-processing" containerID="cri-o://aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28" gracePeriod=25
Nov 03 13:28:17.301906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.301872    5420 generic.go:356] "Generic (PLEG): container finished" podID="2543bc1f-d0ea-4a00-9958-4b20958915b2" containerID="a657a449c789666c432852f5b741167d9af09d4c18c6a708ea81fbb33199108c" exitCode=0
Nov 03 13:28:17.302031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.301994    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln" event={"ID":"2543bc1f-d0ea-4a00-9958-4b20958915b2","Type":"ContainerDied","Data":"a657a449c789666c432852f5b741167d9af09d4c18c6a708ea81fbb33199108c"}
Nov 03 13:28:17.303511 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.303480    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv" event={"ID":"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb","Type":"ContainerStarted","Data":"3f7b1da3cabc8cc2748076701b5ca89d165bdf8999f51fbb01a1e1e02075ff78"}
Nov 03 13:28:17.556159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.556061    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:17.656748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.656678    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:28:17.823131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.823102    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-provider-credentials\") pod \"0de9561a-b039-49f1-a901-d338e7e49820\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") "
Nov 03 13:28:17.823591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.823171    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-routing-overlay-volume\") pod \"0de9561a-b039-49f1-a901-d338e7e49820\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") "
Nov 03 13:28:17.823591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.823269    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-plugins-config-volume\") pod \"0de9561a-b039-49f1-a901-d338e7e49820\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") "
Nov 03 13:28:17.823591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.823324    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-kube-api-access-tts2z\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-kube-api-access-tts2z\") pod \"0de9561a-b039-49f1-a901-d338e7e49820\" (UID: \"0de9561a-b039-49f1-a901-d338e7e49820\") "
Nov 03 13:28:17.823739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.823653    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-routing-overlay-volume" pod "0de9561a-b039-49f1-a901-d338e7e49820" (UID: "0de9561a-b039-49f1-a901-d338e7e49820"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:17.823739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.823691    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-plugins-config-volume" pod "0de9561a-b039-49f1-a901-d338e7e49820" (UID: "0de9561a-b039-49f1-a901-d338e7e49820"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:17.824073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.824052    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:17.824073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.824074    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de9561a-b039-49f1-a901-d338e7e49820-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:17.826183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.826155    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-provider-credentials" pod "0de9561a-b039-49f1-a901-d338e7e49820" (UID: "0de9561a-b039-49f1-a901-d338e7e49820"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:17.826326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.826194    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-kube-api-access-tts2z" pod "0de9561a-b039-49f1-a901-d338e7e49820" (UID: "0de9561a-b039-49f1-a901-d338e7e49820"). InnerVolumeSpecName "kube-api-access-tts2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:17.926045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.925971    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:17.926045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.925998    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tts2z\" (UniqueName: \"kubernetes.io/projected/0de9561a-b039-49f1-a901-d338e7e49820-kube-api-access-tts2z\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:17.994280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:17.994252    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:18.335862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.335782    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:18.340689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.340664    5420 generic.go:356] "Generic (PLEG): container finished" podID="0de9561a-b039-49f1-a901-d338e7e49820" containerID="aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28" exitCode=0
Nov 03 13:28:18.340842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.340711    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"
Nov 03 13:28:18.340842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.340750    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh" event={"ID":"0de9561a-b039-49f1-a901-d338e7e49820","Type":"ContainerDied","Data":"aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28"}
Nov 03 13:28:18.340842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.340809    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh" event={"ID":"0de9561a-b039-49f1-a901-d338e7e49820","Type":"ContainerDied","Data":"39724c5963f9b6d7d51673857f6f66f8b4612921c9237df5d493a77b264dc95d"}
Nov 03 13:28:18.340842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.340836    5420 scope.go:122] "RemoveContainer" containerID="aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28"
Nov 03 13:28:18.344793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.344769    5420 generic.go:356] "Generic (PLEG): container finished" podID="9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb" containerID="86eb11f58a9801244af9966b816e9740f0815bfaa3fd48c519722b4d098561b5" exitCode=0
Nov 03 13:28:18.344942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.344878    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv" event={"ID":"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb","Type":"ContainerDied","Data":"86eb11f58a9801244af9966b816e9740f0815bfaa3fd48c519722b4d098561b5"}
Nov 03 13:28:18.355470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.355425    5420 scope.go:122] "RemoveContainer" containerID="aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28"
Nov 03 13:28:18.355752 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:28:18.355725    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28\": container with ID starting with aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28 not found: ID does not exist" containerID="aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28"
Nov 03 13:28:18.355848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.355756    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28"} err="failed to get container status \"aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28\": rpc error: code = NotFound desc = could not find container \"aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28\": container with ID starting with aa5cff44d37962102cd68158f49f13176fe373ea419c9450c5e4f71c78411a28 not found: ID does not exist"
Nov 03 13:28:18.393418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.393346    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"]
Nov 03 13:28:18.408326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.404622    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b7xtsrh"]
Nov 03 13:28:18.546404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.546382    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:18.650088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.650016    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-xks-ca\") pod \"2543bc1f-d0ea-4a00-9958-4b20958915b2\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") "
Nov 03 13:28:18.650088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.650061    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-openshift-ca\") pod \"2543bc1f-d0ea-4a00-9958-4b20958915b2\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") "
Nov 03 13:28:18.650362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.650308    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2543bc1f-d0ea-4a00-9958-4b20958915b2-kube-api-access-9xzdw\" (UniqueName: \"kubernetes.io/projected/2543bc1f-d0ea-4a00-9958-4b20958915b2-kube-api-access-9xzdw\") pod \"2543bc1f-d0ea-4a00-9958-4b20958915b2\" (UID: \"2543bc1f-d0ea-4a00-9958-4b20958915b2\") "
Nov 03 13:28:18.650609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.650583    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-openshift-ca" pod "2543bc1f-d0ea-4a00-9958-4b20958915b2" (UID: "2543bc1f-d0ea-4a00-9958-4b20958915b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:18.650840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.650823    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:18.652188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.652164    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-xks-ca" pod "2543bc1f-d0ea-4a00-9958-4b20958915b2" (UID: "2543bc1f-d0ea-4a00-9958-4b20958915b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:28:18.652376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.652356    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2543bc1f-d0ea-4a00-9958-4b20958915b2-kube-api-access-9xzdw" pod "2543bc1f-d0ea-4a00-9958-4b20958915b2" (UID: "2543bc1f-d0ea-4a00-9958-4b20958915b2"). InnerVolumeSpecName "kube-api-access-9xzdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:18.752158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.752131    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2543bc1f-d0ea-4a00-9958-4b20958915b2-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:18.752158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:18.752157    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xzdw\" (UniqueName: \"kubernetes.io/projected/2543bc1f-d0ea-4a00-9958-4b20958915b2-kube-api-access-9xzdw\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:19.032316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.032279    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:19.087369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.087341    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de9561a-b039-49f1-a901-d338e7e49820" path="/var/lib/kubelet/pods/0de9561a-b039-49f1-a901-d338e7e49820/volumes"
Nov 03 13:28:19.357406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.357337    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"
Nov 03 13:28:19.357406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.357370    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln" event={"ID":"2543bc1f-d0ea-4a00-9958-4b20958915b2","Type":"ContainerDied","Data":"2f3281fed7c9a7cc76f03d15e4ff1bdb473f3cd57e67f71a7c912219dff884ae"}
Nov 03 13:28:19.357600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.357414    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f3281fed7c9a7cc76f03d15e4ff1bdb473f3cd57e67f71a7c912219dff884ae"
Nov 03 13:28:19.515264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.515230    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:19.563700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.563673    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-openshift-ca\") pod \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") "
Nov 03 13:28:19.563964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.563946    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-xks-ca\") pod \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") "
Nov 03 13:28:19.564026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.563983    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-kube-api-access-v2jbd\" (UniqueName: \"kubernetes.io/projected/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-kube-api-access-v2jbd\") pod \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\" (UID: \"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb\") "
Nov 03 13:28:19.564120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.564091    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-openshift-ca" pod "9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb" (UID: "9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:19.564693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.564666    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:19.566073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.566043    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-xks-ca" pod "9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb" (UID: "9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:28:19.566248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.566200    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-kube-api-access-v2jbd" pod "9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb" (UID: "9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb"). InnerVolumeSpecName "kube-api-access-v2jbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:19.644987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.644895    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:19.666860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.666824    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:19.666860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:19.666854    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jbd\" (UniqueName: \"kubernetes.io/projected/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb-kube-api-access-v2jbd\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:20.371245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:20.371218    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"
Nov 03 13:28:20.371720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:20.371198    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv" event={"ID":"9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb","Type":"ContainerDied","Data":"3f7b1da3cabc8cc2748076701b5ca89d165bdf8999f51fbb01a1e1e02075ff78"}
Nov 03 13:28:20.371720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:20.371334    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f7b1da3cabc8cc2748076701b5ca89d165bdf8999f51fbb01a1e1e02075ff78"
Nov 03 13:28:20.741515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:20.741489    5420 scope.go:122] "RemoveContainer" containerID="3c5b5369727d4be99307c9f0a1872eef5b8ef0b7a0072c92aa09c0661335ee43"
Nov 03 13:28:20.752349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:20.752327    5420 scope.go:122] "RemoveContainer" containerID="1b336f459506efd0ecca35018e684f02ef55531517ac06dc1990a59749c0eaed"
Nov 03 13:28:20.762080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:20.762059    5420 scope.go:122] "RemoveContainer" containerID="f245f63e96fdc25b8e0644df9dac754117719a93433bf5776d65156a4fb2f0a6"
Nov 03 13:28:20.771308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:20.771288    5420 scope.go:122] "RemoveContainer" containerID="b7b958dff4bc61b1431eaa5645e013a5a11f5c8682a0dac3c6daf11964f39bbb"
Nov 03 13:28:20.780619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:20.780598    5420 scope.go:122] "RemoveContainer" containerID="984e5daca7299738db7098243f2419cbcda162deea16b93d2ef2c0a34e2d8925"
Nov 03 13:28:20.844199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:20.844176    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:21.008814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:21.008725    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:21.682480 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:21.682442    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:22.074585 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:22.074554    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:28:22.501853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:22.501816    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:23.339364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:23.339330    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:24.045723 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:24.045691    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"]
Nov 03 13:28:24.056198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:24.056007    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227501c7-rjk9n"]
Nov 03 13:28:24.161144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:24.161110    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:24.732549 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:28:24.732516    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:28:24.952046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:24.952003    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:25.086471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:25.086400    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333" path="/var/lib/kubelet/pods/8eff82c1-72c1-4fb8-8f53-3c1aa2dbf333/volumes"
Nov 03 13:28:25.121236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:25.121178    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:25.916388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:25.916280    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:26.692757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:26.692728    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:26.913271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:26.913233    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:27.086659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.086623    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:27.354288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.354175    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"]
Nov 03 13:28:27.355352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.355321    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2543bc1f-d0ea-4a00-9958-4b20958915b2" containerName="revoke-keys"
Nov 03 13:28:27.355352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.355348    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2543bc1f-d0ea-4a00-9958-4b20958915b2" containerName="revoke-keys"
Nov 03 13:28:27.355578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.355376    5420 cpu_manager.go:414] "containerMap: removing container" podUID="9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb" containerName="revoke-keys"
Nov 03 13:28:27.355578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.355384    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb" containerName="revoke-keys"
Nov 03 13:28:27.355578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.355460    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0de9561a-b039-49f1-a901-d338e7e49820" containerName="payload-processing"
Nov 03 13:28:27.355578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.355468    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de9561a-b039-49f1-a901-d338e7e49820" containerName="payload-processing"
Nov 03 13:28:27.355816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.355640    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de9561a-b039-49f1-a901-d338e7e49820" containerName="payload-processing"
Nov 03 13:28:27.355816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.355657    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2543bc1f-d0ea-4a00-9958-4b20958915b2" containerName="revoke-keys"
Nov 03 13:28:27.355816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.355668    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb" containerName="revoke-keys"
Nov 03 13:28:27.362421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.362395    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.365277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.364925    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"]
Nov 03 13:28:27.368688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.368662    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:28:27.370162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.370138    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:28:27.370334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.370315    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:28:27.370419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.370395    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:28:27.472066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.472036    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhcc\" (UniqueName: \"kubernetes.io/projected/bd045566-c1a6-468a-9a18-f8b92430f049-kube-api-access-qwhcc\") pod \"maas-api-revoke-sub-958a8579-g72n9\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.472279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.472075    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958a8579-g72n9\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.472279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.472226    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958a8579-g72n9\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.521682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.521633    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"]
Nov 03 13:28:27.533625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.533595    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.540985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.540952    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c192b\""
Nov 03 13:28:27.541277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.541235    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-29dd648f7d07\""
Nov 03 13:28:27.541490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.541302    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"openshift-service-ca.crt\""
Nov 03 13:28:27.541635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.541438    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-e2e--cd1da41a-dockercfg-txc88\""
Nov 03 13:28:27.541758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.541538    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"routing-overlay\""
Nov 03 13:28:27.541881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.541623    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"kube-root-ca.crt\""
Nov 03 13:28:27.550463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.550436    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"]
Nov 03 13:28:27.573685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.573651    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.573838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.573700    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958a8579-g72n9\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.573965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.573946    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbvx\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-kube-api-access-rdbvx\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.574031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.573990    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.574107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.574048    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.574107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.574084    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhcc\" (UniqueName: \"kubernetes.io/projected/bd045566-c1a6-468a-9a18-f8b92430f049-kube-api-access-qwhcc\") pod \"maas-api-revoke-sub-958a8579-g72n9\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.574231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.574119    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958a8579-g72n9\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.575131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.575098    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958a8579-g72n9\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.576719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.576692    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958a8579-g72n9\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.585987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.585935    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhcc\" (UniqueName: \"kubernetes.io/projected/bd045566-c1a6-468a-9a18-f8b92430f049-kube-api-access-qwhcc\") pod \"maas-api-revoke-sub-958a8579-g72n9\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.676469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.676369    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.676665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.676646    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbvx\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-kube-api-access-rdbvx\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.676740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.676681    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.676740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.676722    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.677976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.677194    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.677976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.677940    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.679382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.679358    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.680321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.680295    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:27.687850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.687829    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbvx\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-kube-api-access-rdbvx\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.858258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.858225    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:27.984826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.984795    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:27.996993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:27.996910    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"]
Nov 03 13:28:28.178270 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:28:28.178237    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode540a1a1_10de_45c5_a932_dc9f1e9cf268.slice/crio-c0e0253133f63a5048740515f2d293f733cb82d55066f9be655d3e20a320daa9 WatchSource:0}: Error finding container c0e0253133f63a5048740515f2d293f733cb82d55066f9be655d3e20a320daa9: Status 404 returned error can't find the container with id c0e0253133f63a5048740515f2d293f733cb82d55066f9be655d3e20a320daa9
Nov 03 13:28:28.179966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.179944    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"]
Nov 03 13:28:28.434023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.433977    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:28.493273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.493192    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d" event={"ID":"e540a1a1-10de-45c5-a932-dc9f1e9cf268","Type":"ContainerStarted","Data":"a0cfa44f2d2d4113cb5e324dd3ef9c0e224a83ee5e8989db04e7a73edc0a1c5b"}
Nov 03 13:28:28.493468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.493295    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d" event={"ID":"e540a1a1-10de-45c5-a932-dc9f1e9cf268","Type":"ContainerStarted","Data":"c0e0253133f63a5048740515f2d293f733cb82d55066f9be655d3e20a320daa9"}
Nov 03 13:28:28.495494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.495466    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9" event={"ID":"bd045566-c1a6-468a-9a18-f8b92430f049","Type":"ContainerStarted","Data":"85378d50b5c0d466315481245d10cd009684557e87d08d6e58edd2c41ab22b92"}
Nov 03 13:28:28.495636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.495508    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9" event={"ID":"bd045566-c1a6-468a-9a18-f8b92430f049","Type":"ContainerStarted","Data":"1375db9c3534853cf2f774fedcb49e934b6511cec840bafad4db0c70a5c88b9f"}
Nov 03 13:28:28.517053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.516996    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d" podStartSLOduration=1.516978124 podStartE2EDuration="1.516978124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:28.513643306 +0000 UTC m=+1149.591408359" watchObservedRunningTime="2026-11-03 13:28:28.516978124 +0000 UTC m=+1149.594743152"
Nov 03 13:28:28.536744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.536694    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9" podStartSLOduration=1.536679555 podStartE2EDuration="1.536679555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:28.533705467 +0000 UTC m=+1149.611470503" watchObservedRunningTime="2026-11-03 13:28:28.536679555 +0000 UTC m=+1149.614444609"
Nov 03 13:28:28.598059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.598026    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:28.764121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:28.764047    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:29.440873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:29.440777    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:29.514886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:29.514847    5420 generic.go:356] "Generic (PLEG): container finished" podID="bd045566-c1a6-468a-9a18-f8b92430f049" containerID="85378d50b5c0d466315481245d10cd009684557e87d08d6e58edd2c41ab22b92" exitCode=0
Nov 03 13:28:29.515086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:29.514965    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9" event={"ID":"bd045566-c1a6-468a-9a18-f8b92430f049","Type":"ContainerDied","Data":"85378d50b5c0d466315481245d10cd009684557e87d08d6e58edd2c41ab22b92"}
Nov 03 13:28:29.515375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:29.515340    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:30.129858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.129824    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:30.696895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.696872    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:30.732906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.732881    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd045566-c1a6-468a-9a18-f8b92430f049-kube-api-access-qwhcc\" (UniqueName: \"kubernetes.io/projected/bd045566-c1a6-468a-9a18-f8b92430f049-kube-api-access-qwhcc\") pod \"bd045566-c1a6-468a-9a18-f8b92430f049\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") "
Nov 03 13:28:30.733072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.732993    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-xks-ca\") pod \"bd045566-c1a6-468a-9a18-f8b92430f049\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") "
Nov 03 13:28:30.733141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.733122    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-openshift-ca\") pod \"bd045566-c1a6-468a-9a18-f8b92430f049\" (UID: \"bd045566-c1a6-468a-9a18-f8b92430f049\") "
Nov 03 13:28:30.733477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.733448    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-openshift-ca" pod "bd045566-c1a6-468a-9a18-f8b92430f049" (UID: "bd045566-c1a6-468a-9a18-f8b92430f049"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:30.734014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.733992    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:30.735117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.735089    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-xks-ca" pod "bd045566-c1a6-468a-9a18-f8b92430f049" (UID: "bd045566-c1a6-468a-9a18-f8b92430f049"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:28:30.735288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.735233    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd045566-c1a6-468a-9a18-f8b92430f049-kube-api-access-qwhcc" pod "bd045566-c1a6-468a-9a18-f8b92430f049" (UID: "bd045566-c1a6-468a-9a18-f8b92430f049"). InnerVolumeSpecName "kube-api-access-qwhcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:30.835775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.835698    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwhcc\" (UniqueName: \"kubernetes.io/projected/bd045566-c1a6-468a-9a18-f8b92430f049-kube-api-access-qwhcc\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:30.835775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.835725    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd045566-c1a6-468a-9a18-f8b92430f049-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:30.848474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:30.848431    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:31.534554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:31.534522    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"
Nov 03 13:28:31.534724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:31.534550    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9" event={"ID":"bd045566-c1a6-468a-9a18-f8b92430f049","Type":"ContainerDied","Data":"1375db9c3534853cf2f774fedcb49e934b6511cec840bafad4db0c70a5c88b9f"}
Nov 03 13:28:31.534724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:31.534596    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1375db9c3534853cf2f774fedcb49e934b6511cec840bafad4db0c70a5c88b9f"
Nov 03 13:28:31.690142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:31.690109    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:32.528260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:32.527503    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:33.231576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:33.231538    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:33.780751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:33.780722    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:33.982597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:33.982565    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:34.065366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:34.065286    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"]
Nov 03 13:28:34.070711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:34.070683    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef15d972-vmndv"]
Nov 03 13:28:34.226691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:34.226647    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:34.424132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:34.424032    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:35.050357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:35.050308    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"]
Nov 03 13:28:35.056192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:35.056160    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71060761-vwjll"]
Nov 03 13:28:35.085133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:35.085099    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:35.107270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:35.107234    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6895e9-17e7-4555-90a9-4552dc4e2628" path="/var/lib/kubelet/pods/7f6895e9-17e7-4555-90a9-4552dc4e2628/volumes"
Nov 03 13:28:35.108335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:35.108308    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb" path="/var/lib/kubelet/pods/9e6d5989-6ed4-448a-a3bd-fd53bf29c3eb/volumes"
Nov 03 13:28:35.314375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:35.314288    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:35.679237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:35.679142    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:36.109733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:36.109697    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:36.288401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:36.288372    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:37.372573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:37.372546    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:38.037012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:38.036977    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"]
Nov 03 13:28:38.041623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:38.041602    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770053ed-z2ww9"]
Nov 03 13:28:38.245339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:38.245278    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:38.425890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:38.425801    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"]
Nov 03 13:28:38.428902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:38.428881    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:38.472408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:38.472374    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:38.688286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:38.685839    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:39.088419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:39.088391    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c51a890-a003-4b94-9d1e-8517bbe774e5" path="/var/lib/kubelet/pods/9c51a890-a003-4b94-9d1e-8517bbe774e5/volumes"
Nov 03 13:28:39.133460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:39.133428    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:39.330392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:39.330360    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:39.813759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:39.813728    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:39.944581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:39.944549    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:40.477333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:40.476823    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:41.062972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:41.062930    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:41.283690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:41.283655    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:41.474181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:41.474146    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:41.735784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:41.735695    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:41.916078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:41.916035    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:42.230864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:42.230834    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:42.772775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:42.772728    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:42.971359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:42.971325    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:43.408367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.408330    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:43.427002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.426966    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d" podUID="e540a1a1-10de-45c5-a932-dc9f1e9cf268" containerName="payload-processing" containerID="cri-o://a0cfa44f2d2d4113cb5e324dd3ef9c0e224a83ee5e8989db04e7a73edc0a1c5b" gracePeriod=25
Nov 03 13:28:43.688722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.688640    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:43.690200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.690175    5420 generic.go:356] "Generic (PLEG): container finished" podID="e540a1a1-10de-45c5-a932-dc9f1e9cf268" containerID="a0cfa44f2d2d4113cb5e324dd3ef9c0e224a83ee5e8989db04e7a73edc0a1c5b" exitCode=0
Nov 03 13:28:43.690360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.690240    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d" event={"ID":"e540a1a1-10de-45c5-a932-dc9f1e9cf268","Type":"ContainerDied","Data":"a0cfa44f2d2d4113cb5e324dd3ef9c0e224a83ee5e8989db04e7a73edc0a1c5b"}
Nov 03 13:28:43.733196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.733168    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:43.824820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.824785    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-provider-credentials\") pod \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") "
Nov 03 13:28:43.825024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.824914    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-plugins-config-volume\") pod \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") "
Nov 03 13:28:43.825111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.825053    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-kube-api-access-rdbvx\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-kube-api-access-rdbvx\") pod \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") "
Nov 03 13:28:43.825225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.825184    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-routing-overlay-volume\") pod \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\" (UID: \"e540a1a1-10de-45c5-a932-dc9f1e9cf268\") "
Nov 03 13:28:43.825356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.825329    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-plugins-config-volume" pod "e540a1a1-10de-45c5-a932-dc9f1e9cf268" (UID: "e540a1a1-10de-45c5-a932-dc9f1e9cf268"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:43.825546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.825521    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-routing-overlay-volume" pod "e540a1a1-10de-45c5-a932-dc9f1e9cf268" (UID: "e540a1a1-10de-45c5-a932-dc9f1e9cf268"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:28:43.825986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.825968    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:43.826088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.825992    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e540a1a1-10de-45c5-a932-dc9f1e9cf268-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:43.827485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.827453    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-kube-api-access-rdbvx" pod "e540a1a1-10de-45c5-a932-dc9f1e9cf268" (UID: "e540a1a1-10de-45c5-a932-dc9f1e9cf268"). InnerVolumeSpecName "kube-api-access-rdbvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:43.827575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.827510    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-provider-credentials" pod "e540a1a1-10de-45c5-a932-dc9f1e9cf268" (UID: "e540a1a1-10de-45c5-a932-dc9f1e9cf268"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:28:43.934267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.932625    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:43.934267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:43.932663    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdbvx\" (UniqueName: \"kubernetes.io/projected/e540a1a1-10de-45c5-a932-dc9f1e9cf268-kube-api-access-rdbvx\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:28:44.276902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:44.276874    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:44.700384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:44.700314    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"
Nov 03 13:28:44.700384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:44.700363    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d" event={"ID":"e540a1a1-10de-45c5-a932-dc9f1e9cf268","Type":"ContainerDied","Data":"c0e0253133f63a5048740515f2d293f733cb82d55066f9be655d3e20a320daa9"}
Nov 03 13:28:44.700815 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:44.700418    5420 scope.go:122] "RemoveContainer" containerID="a0cfa44f2d2d4113cb5e324dd3ef9c0e224a83ee5e8989db04e7a73edc0a1c5b"
Nov 03 13:28:44.729783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:44.729753    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"]
Nov 03 13:28:44.735059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:44.735037    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7866k2p7d"]
Nov 03 13:28:44.918376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:44.918344    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:45.087360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:45.087326    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e540a1a1-10de-45c5-a932-dc9f1e9cf268" path="/var/lib/kubelet/pods/e540a1a1-10de-45c5-a932-dc9f1e9cf268/volumes"
Nov 03 13:28:45.102102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:45.102035    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:45.286794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:45.286766    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:45.464043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:45.464010    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:46.032423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:46.032374    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:46.232809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:46.232778    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:46.439427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:46.438711    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:46.915386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:46.915358    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:47.649174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:47.649120    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:48.293309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:48.293093    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:49.026891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:49.026858    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"]
Nov 03 13:28:49.033894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:49.033867    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40055367-g8k28"]
Nov 03 13:28:49.037937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:49.037921    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"]
Nov 03 13:28:49.041754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:49.041734    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd9cf826-wkgps"]
Nov 03 13:28:49.097063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:49.097031    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373412bc-e342-4779-acac-97e4fe59f367" path="/var/lib/kubelet/pods/373412bc-e342-4779-acac-97e4fe59f367/volumes"
Nov 03 13:28:49.097519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:49.097503    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9adc27ef-51fe-40fd-9375-d97ef8cef7f5" path="/var/lib/kubelet/pods/9adc27ef-51fe-40fd-9375-d97ef8cef7f5/volumes"
Nov 03 13:28:49.231860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:49.231831    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:49.869238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:49.868608    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:50.859173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:50.859133    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:51.543854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:51.543820    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:52.330164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:52.330086    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:52.945904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:52.945860    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:53.686664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:53.686629    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:53.885048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:53.885005    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:54.058128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:54.058097    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:54.647294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:54.647091    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:54.758942 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:28:54.758904    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:28:54.846073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:54.846026    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:54.994180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:54.994148    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:55.707227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:55.707183    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:55.875952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:55.875918    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:56.539485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:56.539392    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:57.250632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:57.250595    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:57.941516 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:57.941470    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:58.539805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:58.539727    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:59.158377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.158341    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:59.290372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.290330    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:28:59.290556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.290393    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:28:59.821720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.821618    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:59.961707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.961674    5420 generic.go:356] "Generic (PLEG): container finished" podID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerID="90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98" exitCode=0
Nov 03 13:28:59.961912 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.961790    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" event={"ID":"0b26dd89-8856-4b20-b5be-62a4abf9c9a6","Type":"ContainerDied","Data":"90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98"}
Nov 03 13:28:59.962477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.962455    5420 scope.go:122] "RemoveContainer" containerID="90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98"
Nov 03 13:28:59.964660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.964604    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:28:59.964777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.964690    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:28:59.964777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:28:59.964735    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:00.529693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.529660    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:00.583274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.579951    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"]
Nov 03 13:29:00.585006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.584964    5420 cpu_manager.go:414] "containerMap: removing container" podUID="bd045566-c1a6-468a-9a18-f8b92430f049" containerName="revoke-keys"
Nov 03 13:29:00.585190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.585175    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd045566-c1a6-468a-9a18-f8b92430f049" containerName="revoke-keys"
Nov 03 13:29:00.585357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.585343    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e540a1a1-10de-45c5-a932-dc9f1e9cf268" containerName="payload-processing"
Nov 03 13:29:00.585452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.585440    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e540a1a1-10de-45c5-a932-dc9f1e9cf268" containerName="payload-processing"
Nov 03 13:29:00.585744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.585727    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd045566-c1a6-468a-9a18-f8b92430f049" containerName="revoke-keys"
Nov 03 13:29:00.585863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.585853    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e540a1a1-10de-45c5-a932-dc9f1e9cf268" containerName="payload-processing"
Nov 03 13:29:00.608629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.608598    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"]
Nov 03 13:29:00.608927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.608911    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.613969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.613873    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:29:00.614196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.613985    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:29:00.614196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.614126    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:29:00.619032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.619005    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:29:00.694126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.694088    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkgr\" (UniqueName: \"kubernetes.io/projected/88edf335-9304-429c-8607-f2d8b7b446f5-kube-api-access-xbkgr\") pod \"maas-api-revoke-sub-9a4eb40d-lwrt6\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.694126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.694128    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a4eb40d-lwrt6\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.694346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.694328    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a4eb40d-lwrt6\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.795622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.795531    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkgr\" (UniqueName: \"kubernetes.io/projected/88edf335-9304-429c-8607-f2d8b7b446f5-kube-api-access-xbkgr\") pod \"maas-api-revoke-sub-9a4eb40d-lwrt6\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.795622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.795578    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a4eb40d-lwrt6\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.795881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.795745    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a4eb40d-lwrt6\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.796452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.796428    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a4eb40d-lwrt6\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.798125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.798107    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a4eb40d-lwrt6\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.803999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.803976    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkgr\" (UniqueName: \"kubernetes.io/projected/88edf335-9304-429c-8607-f2d8b7b446f5-kube-api-access-xbkgr\") pod \"maas-api-revoke-sub-9a4eb40d-lwrt6\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.960701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.960667    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:00.975661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:00.975632    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" event={"ID":"0b26dd89-8856-4b20-b5be-62a4abf9c9a6","Type":"ContainerStarted","Data":"ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8"}
Nov 03 13:29:01.219575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:01.219539    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:29:01.219746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:01.219707    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 03 13:29:01.219823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:01.219750    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 03 13:29:01.238432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:01.235147    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:01.281857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:01.281816    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"]
Nov 03 13:29:01.830291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:01.829231    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:01.985239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:01.985188    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6" event={"ID":"88edf335-9304-429c-8607-f2d8b7b446f5","Type":"ContainerStarted","Data":"a53bb3250ee1a63724cadca39fd52b9c43bfb7d0848eeba0332e8eeaf2130ba4"}
Nov 03 13:29:01.985728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:01.985253    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6" event={"ID":"88edf335-9304-429c-8607-f2d8b7b446f5","Type":"ContainerStarted","Data":"6d5a01606b2c5476047a86eeaf46a3c43bd97dbfb64bba659a398113fc30ee03"}
Nov 03 13:29:02.016494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:02.016437    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6" podStartSLOduration=2.016421628 podStartE2EDuration="2.016421628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:02.01303535 +0000 UTC m=+1183.090800368" watchObservedRunningTime="2026-11-03 13:29:02.016421628 +0000 UTC m=+1183.094186654"
Nov 03 13:29:02.127290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:02.120345    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:02.219704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:02.219668    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 03 13:29:02.219879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:02.219725    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 03 13:29:02.410010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:02.409924    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:02.945741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:02.945712    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:03.004977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:03.004930    5420 generic.go:356] "Generic (PLEG): container finished" podID="88edf335-9304-429c-8607-f2d8b7b446f5" containerID="a53bb3250ee1a63724cadca39fd52b9c43bfb7d0848eeba0332e8eeaf2130ba4" exitCode=0
Nov 03 13:29:03.005557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:03.005102    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6" event={"ID":"88edf335-9304-429c-8607-f2d8b7b446f5","Type":"ContainerDied","Data":"a53bb3250ee1a63724cadca39fd52b9c43bfb7d0848eeba0332e8eeaf2130ba4"}
Nov 03 13:29:03.219597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:03.219568    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 03 13:29:03.219830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:03.219809    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 03 13:29:03.632972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:03.632869    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:04.210993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.210958    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:04.219506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.219479    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 03 13:29:04.219632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.219523    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 03 13:29:04.305063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.305037    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:04.396733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.396702    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88edf335-9304-429c-8607-f2d8b7b446f5-kube-api-access-xbkgr\" (UniqueName: \"kubernetes.io/projected/88edf335-9304-429c-8607-f2d8b7b446f5-kube-api-access-xbkgr\") pod \"88edf335-9304-429c-8607-f2d8b7b446f5\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") "
Nov 03 13:29:04.396966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.396946    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-xks-ca\") pod \"88edf335-9304-429c-8607-f2d8b7b446f5\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") "
Nov 03 13:29:04.397086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.396998    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-openshift-ca\") pod \"88edf335-9304-429c-8607-f2d8b7b446f5\" (UID: \"88edf335-9304-429c-8607-f2d8b7b446f5\") "
Nov 03 13:29:04.397417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.397388    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-openshift-ca" pod "88edf335-9304-429c-8607-f2d8b7b446f5" (UID: "88edf335-9304-429c-8607-f2d8b7b446f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:04.397772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.397755    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:04.398956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.398929    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88edf335-9304-429c-8607-f2d8b7b446f5-kube-api-access-xbkgr" pod "88edf335-9304-429c-8607-f2d8b7b446f5" (UID: "88edf335-9304-429c-8607-f2d8b7b446f5"). InnerVolumeSpecName "kube-api-access-xbkgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:04.399024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.399008    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-xks-ca" pod "88edf335-9304-429c-8607-f2d8b7b446f5" (UID: "88edf335-9304-429c-8607-f2d8b7b446f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:29:04.499743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.499712    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88edf335-9304-429c-8607-f2d8b7b446f5-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:04.499743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.499742    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbkgr\" (UniqueName: \"kubernetes.io/projected/88edf335-9304-429c-8607-f2d8b7b446f5-kube-api-access-xbkgr\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:04.831683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:04.831647    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:05.044234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:05.043372    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6" event={"ID":"88edf335-9304-429c-8607-f2d8b7b446f5","Type":"ContainerDied","Data":"6d5a01606b2c5476047a86eeaf46a3c43bd97dbfb64bba659a398113fc30ee03"}
Nov 03 13:29:05.044234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:05.043435    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d5a01606b2c5476047a86eeaf46a3c43bd97dbfb64bba659a398113fc30ee03"
Nov 03 13:29:05.044234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:05.043505    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"
Nov 03 13:29:05.242259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:05.230327    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 03 13:29:05.242259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:05.230398    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 03 13:29:05.501290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:05.498125    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:06.171714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:06.171659    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:06.224494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:06.224460    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:29:06.225050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:06.225018    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:29:06.225788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:06.225769    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:29:06.675643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:06.675610    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:29:06.807655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:06.807625    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:07.535226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:07.535193    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:08.031079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:08.031046    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"]
Nov 03 13:29:08.035770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:08.035747    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a4c5a1-zxd2d"]
Nov 03 13:29:08.436965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:08.436884    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:09.123734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:09.123696    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f596775-f524-4f4c-8ba9-473432865682" path="/var/lib/kubelet/pods/1f596775-f524-4f4c-8ba9-473432865682/volumes"
Nov 03 13:29:09.157012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:09.156977    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:09.593798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:09.593759    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:09.955345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:09.955314    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:11.003040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.003003    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"]
Nov 03 13:29:11.003780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.003763    5420 cpu_manager.go:414] "containerMap: removing container" podUID="88edf335-9304-429c-8607-f2d8b7b446f5" containerName="revoke-keys"
Nov 03 13:29:11.003831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.003781    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88edf335-9304-429c-8607-f2d8b7b446f5" containerName="revoke-keys"
Nov 03 13:29:11.003888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.003879    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="88edf335-9304-429c-8607-f2d8b7b446f5" containerName="revoke-keys"
Nov 03 13:29:11.007944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.007923    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.022638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.022602    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:29:11.022784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.022677    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:29:11.025303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.025278    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:29:11.025512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.025489    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:29:11.025890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.025876    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"]
Nov 03 13:29:11.096228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.096190    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:11.123528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.123495    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlnbw\" (UniqueName: \"kubernetes.io/projected/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-kube-api-access-vlnbw\") pod \"maas-api-revoke-sub-63e21aca-qpgtq\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.123726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.123640    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e21aca-qpgtq\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.123810 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.123786    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e21aca-qpgtq\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.225049 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.225004    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e21aca-qpgtq\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.225321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.225300    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlnbw\" (UniqueName: \"kubernetes.io/projected/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-kube-api-access-vlnbw\") pod \"maas-api-revoke-sub-63e21aca-qpgtq\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.225464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.225440    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e21aca-qpgtq\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.226082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.226060    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e21aca-qpgtq\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.227541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.227518    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e21aca-qpgtq\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.234413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.234388    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlnbw\" (UniqueName: \"kubernetes.io/projected/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-kube-api-access-vlnbw\") pod \"maas-api-revoke-sub-63e21aca-qpgtq\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.324126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.324036    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:11.620244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.620190    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"]
Nov 03 13:29:11.620737 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:29:11.620708    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3fcace_0be4_4c8b_ac35_f83cf8f25a4b.slice/crio-b86946075939d7c25716701ddb2631f09eaf10ff075fabf96b3d7c70b8d2bc0a WatchSource:0}: Error finding container b86946075939d7c25716701ddb2631f09eaf10ff075fabf96b3d7c70b8d2bc0a: Status 404 returned error can't find the container with id b86946075939d7c25716701ddb2631f09eaf10ff075fabf96b3d7c70b8d2bc0a
Nov 03 13:29:11.905265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:11.905139    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:12.151319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:12.151272    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq" event={"ID":"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b","Type":"ContainerStarted","Data":"88481cf8b016e585d0662218d97efce3a141c2c62570276d3c2b1ef7ca8a6ebe"}
Nov 03 13:29:12.151775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:12.151332    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq" event={"ID":"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b","Type":"ContainerStarted","Data":"b86946075939d7c25716701ddb2631f09eaf10ff075fabf96b3d7c70b8d2bc0a"}
Nov 03 13:29:12.174589 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:12.174488    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq" podStartSLOduration=2.174472631 podStartE2EDuration="2.174472631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:12.170957385 +0000 UTC m=+1193.248722414" watchObservedRunningTime="2026-11-03 13:29:12.174472631 +0000 UTC m=+1193.252237702"
Nov 03 13:29:12.572009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:12.571966    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:13.164866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.164825    5420 generic.go:356] "Generic (PLEG): container finished" podID="7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b" containerID="88481cf8b016e585d0662218d97efce3a141c2c62570276d3c2b1ef7ca8a6ebe" exitCode=0
Nov 03 13:29:13.164866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.164861    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq" event={"ID":"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b","Type":"ContainerDied","Data":"88481cf8b016e585d0662218d97efce3a141c2c62570276d3c2b1ef7ca8a6ebe"}
Nov 03 13:29:13.221760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.221299    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:13.502537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.502495    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"]
Nov 03 13:29:13.508441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.508410    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.523428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.523397    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"]
Nov 03 13:29:13.680744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.680699    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsbw\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-kube-api-access-mhsbw\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.680908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.680854    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.680989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.680945    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.681069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.681046    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.681125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.681105    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.681299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.681277    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.681381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.681341    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.681452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.681441    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.681509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.681493    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.782847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.782751    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.782873    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.782922    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.782960    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsbw\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-kube-api-access-mhsbw\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783049    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783090    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783172    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783249    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783256    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783379    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783527    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783701    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783734    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.783992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.783972    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.785107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.785089    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.785707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.785689    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.796718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.796578    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.805713 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.800623    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsbw\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-kube-api-access-mhsbw\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:13.821159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.821126    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:13.829492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:13.829460    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:14.031202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.031168    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"]
Nov 03 13:29:14.038439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.038379    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7228dc7-vxb6p"]
Nov 03 13:29:14.079013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.078984    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:14.136544 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:29:14.136513    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777a0f06_59b2_42ba_b18f_77c6c351a7cd.slice/crio-ae05f34def2f0b4cda8d4d1182c50dc45586eb9fceb755321e3c9fac45f9b3d1 WatchSource:0}: Error finding container ae05f34def2f0b4cda8d4d1182c50dc45586eb9fceb755321e3c9fac45f9b3d1: Status 404 returned error can't find the container with id ae05f34def2f0b4cda8d4d1182c50dc45586eb9fceb755321e3c9fac45f9b3d1
Nov 03 13:29:14.137106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.137083    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"]
Nov 03 13:29:14.139385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.139349    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:14.139478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.139420    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:14.139478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.139446    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:14.175927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.175889    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" event={"ID":"777a0f06-59b2-42ba-b18f-77c6c351a7cd","Type":"ContainerStarted","Data":"ae05f34def2f0b4cda8d4d1182c50dc45586eb9fceb755321e3c9fac45f9b3d1"}
Nov 03 13:29:14.220724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.220700    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:14.337083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.337061    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:14.505241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.501843    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-kube-api-access-vlnbw\" (UniqueName: \"kubernetes.io/projected/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-kube-api-access-vlnbw\") pod \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") "
Nov 03 13:29:14.505241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.502090    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-xks-ca\") pod \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") "
Nov 03 13:29:14.505241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.502134    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-openshift-ca\") pod \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\" (UID: \"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b\") "
Nov 03 13:29:14.529235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.512423    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-openshift-ca" pod "7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b" (UID: "7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:14.585684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.585600    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-kube-api-access-vlnbw" pod "7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b" (UID: "7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b"). InnerVolumeSpecName "kube-api-access-vlnbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:14.625318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.618606    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlnbw\" (UniqueName: \"kubernetes.io/projected/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-kube-api-access-vlnbw\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:14.625318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.618646    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:14.630266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.628010    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-xks-ca" pod "7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b" (UID: "7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:29:14.723260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:14.720078    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:15.096220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.096173    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4bc80b9-15ad-4c81-b1fe-48a61b4d9315" path="/var/lib/kubelet/pods/c4bc80b9-15ad-4c81-b1fe-48a61b4d9315/volumes"
Nov 03 13:29:15.187375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.187345    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"
Nov 03 13:29:15.187375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.187365    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq" event={"ID":"7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b","Type":"ContainerDied","Data":"b86946075939d7c25716701ddb2631f09eaf10ff075fabf96b3d7c70b8d2bc0a"}
Nov 03 13:29:15.187844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.187414    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b86946075939d7c25716701ddb2631f09eaf10ff075fabf96b3d7c70b8d2bc0a"
Nov 03 13:29:15.189560 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.189468    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" event={"ID":"777a0f06-59b2-42ba-b18f-77c6c351a7cd","Type":"ContainerStarted","Data":"fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f"}
Nov 03 13:29:15.259508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.259142    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:15.263412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.263357    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" podStartSLOduration=2.263338226 podStartE2EDuration="2.263338226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:15.22900907 +0000 UTC m=+1196.306774097" watchObservedRunningTime="2026-11-03 13:29:15.263338226 +0000 UTC m=+1196.341103254"
Nov 03 13:29:15.841276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.841009    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:15.848237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.843856    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 13:29:15.848237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.843921    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 13:29:15.966236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:15.963230    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:16.154338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:16.154053    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:16.297468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:16.297438    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:16.793150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:16.787704    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:16.838237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:16.837127    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 13:29:16.838237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:16.837184    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 13:29:17.489681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:17.484603    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:17.843294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:17.836157    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 13:29:17.843294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:17.836269    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 13:29:18.037403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:18.037366    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:18.052427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:18.051561    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"]
Nov 03 13:29:18.057187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:18.057156    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b029f379-tnxlc"]
Nov 03 13:29:18.181723 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:18.181646    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:18.832031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:18.831565    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 13:29:18.832031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:18.831639    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 13:29:18.844288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:18.844158    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:19.088321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:19.088249    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a040a8ac-bd8c-4078-b04d-c398e1f6ba9b" path="/var/lib/kubelet/pods/a040a8ac-bd8c-4078-b04d-c398e1f6ba9b/volumes"
Nov 03 13:29:19.319529 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:19.319495    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:29:19.350799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:19.350708    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:29:19.745794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:19.745754    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:19.834154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:19.834119    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 13:29:19.834844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:19.834808    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 13:29:20.363596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.363563    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:20.498329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.498280    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:20.843561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.834313    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 03 13:29:20.843561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.834374    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 03 13:29:20.847105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.847061    5420 scope.go:122] "RemoveContainer" containerID="f13e199d1fc85065f7fcc90ebb021e2e8522f675ce31a78b7c346c2c9f5094d9"
Nov 03 13:29:20.861079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.861055    5420 scope.go:122] "RemoveContainer" containerID="d8792e2ac3bb09737f20c018259cd3582e893b30f5ba5abbf3568ad9386f0ac9"
Nov 03 13:29:20.872404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.872381    5420 scope.go:122] "RemoveContainer" containerID="267104f3ddc52fb4ad230a73f37cd322cf867149a4abdb9797ed5a3782ff4618"
Nov 03 13:29:20.883010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.882991    5420 scope.go:122] "RemoveContainer" containerID="6f5d3594c18cb5ca419a59b9de4adb1c1466768df3a19db1553f06facfa464d5"
Nov 03 13:29:20.892457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.892437    5420 scope.go:122] "RemoveContainer" containerID="21b9f534af4e5a10e3ba238231118c53e9eb8aa148f823dd99ea3f9721c31987"
Nov 03 13:29:20.901588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.901569    5420 scope.go:122] "RemoveContainer" containerID="ab63aceefb0345c11b6a247fec71d7033d9c9f8597d86f97e498e96ca393d345"
Nov 03 13:29:20.910727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.910705    5420 scope.go:122] "RemoveContainer" containerID="792873828a008109d9ad5699b193abb2daac9c917c23dce403062d3edae72787"
Nov 03 13:29:20.919940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.919922    5420 scope.go:122] "RemoveContainer" containerID="e020f41ae2d2cdf24c15700124366fc2ba5076529bea43b2ada5f6990648cbad"
Nov 03 13:29:20.949256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:20.949198    5420 scope.go:122] "RemoveContainer" containerID="d8e1d79772c3af2fe51d8e0c59f4640a435351e83d40cea09493761b15242c7a"
Nov 03 13:29:21.145881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:21.145782    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:21.834849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:21.834816    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:21.835447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:21.835421    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:21.836300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:21.836284    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:29:21.898139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:21.898107    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:22.625665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:22.625635    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:22.909245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:22.909142    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:23.107054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:23.107026    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:23.239801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:23.239765    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:23.454393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:23.454364    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:24.041203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:24.041151    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"]
Nov 03 13:29:24.045585 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:24.045557    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:24.051277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:24.051252    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a31fe6b2-4l99j"]
Nov 03 13:29:24.722027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:24.721995    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:24.744036 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:29:24.743994    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:29:25.086158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.086086    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa733fd-9c23-4cd7-9998-6413aec87cdb" path="/var/lib/kubelet/pods/1fa733fd-9c23-4cd7-9998-6413aec87cdb/volumes"
Nov 03 13:29:25.407307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.405766    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:25.737709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.737641    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"]
Nov 03 13:29:25.738747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.738722    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b" containerName="revoke-keys"
Nov 03 13:29:25.738747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.738746    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b" containerName="revoke-keys"
Nov 03 13:29:25.738977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.738963    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b" containerName="revoke-keys"
Nov 03 13:29:25.745267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.745245    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.757491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.757450    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"]
Nov 03 13:29:25.763643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.763618    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"]
Nov 03 13:29:25.768052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.767406    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.768052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.767465    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.768052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.767560    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npnv\" (UniqueName: \"kubernetes.io/projected/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-kube-api-access-5npnv\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.768052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.767714    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.768052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.767788    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.768052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.767824    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.768052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.767961    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.768509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.768060    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.768509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.768116    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.768509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.768301    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.799384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.799352    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"]
Nov 03 13:29:25.869755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.869723    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5762c366-b31f-488b-997b-f4ba1a053178-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.869755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.869757    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.870004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.869778    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.870004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.869923    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.869964    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870010    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.870271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870062    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870127    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgszn\" (UniqueName: \"kubernetes.io/projected/5762c366-b31f-488b-997b-f4ba1a053178-kube-api-access-rgszn\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.870271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870171    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870270    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870410    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5762c366-b31f-488b-997b-f4ba1a053178-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.870486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870459    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870462    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870484    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870520    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870559    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870640    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npnv\" (UniqueName: \"kubernetes.io/projected/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-kube-api-access-5npnv\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870645    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.870761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870681    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.870761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870707    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.870761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870742    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.871016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870830    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5762c366-b31f-488b-997b-f4ba1a053178-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.871016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.870876    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.872741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.872713    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.873421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.873399    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.879730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.879706    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.879968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.879944    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npnv\" (UniqueName: \"kubernetes.io/projected/2ae9c04b-23b3-4ea2-baf0-c8940a31044a-kube-api-access-5npnv\") pod \"maas-default-gateway-openshift-default-84587756dc-xs5rk\" (UID: \"2ae9c04b-23b3-4ea2-baf0-c8940a31044a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:25.973228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.972875    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5762c366-b31f-488b-997b-f4ba1a053178-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973044    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973066    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973141    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5762c366-b31f-488b-997b-f4ba1a053178-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973169    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5762c366-b31f-488b-997b-f4ba1a053178-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973197    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973272    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973359    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973447    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgszn\" (UniqueName: \"kubernetes.io/projected/5762c366-b31f-488b-997b-f4ba1a053178-kube-api-access-rgszn\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973511    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.973897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.973729    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5762c366-b31f-488b-997b-f4ba1a053178-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.975665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.974042    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.975665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.974722    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.975665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.974763    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.978147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.977606    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5762c366-b31f-488b-997b-f4ba1a053178-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.979648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.979617    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5762c366-b31f-488b-997b-f4ba1a053178-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.983084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.983060    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgszn\" (UniqueName: \"kubernetes.io/projected/5762c366-b31f-488b-997b-f4ba1a053178-kube-api-access-rgszn\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:25.986450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:25.986427    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5762c366-b31f-488b-997b-f4ba1a053178-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-tb9j4\" (UID: \"5762c366-b31f-488b-997b-f4ba1a053178\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:26.066959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.066892    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:26.127233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.125140    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:26.159248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.146310    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:26.380659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.380573    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:26.450931 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:29:26.450901    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae9c04b_23b3_4ea2_baf0_c8940a31044a.slice/crio-3d706de228656c7686b8534c09bdd4d3c79c6b18d42dde66eb7ddd0ea427981c WatchSource:0}: Error finding container 3d706de228656c7686b8534c09bdd4d3c79c6b18d42dde66eb7ddd0ea427981c: Status 404 returned error can't find the container with id 3d706de228656c7686b8534c09bdd4d3c79c6b18d42dde66eb7ddd0ea427981c
Nov 03 13:29:26.451084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.451021    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"]
Nov 03 13:29:26.453709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.453670    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:26.453824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.453751    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:26.453824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.453800    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:26.507676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.507647    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"]
Nov 03 13:29:26.512312 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:29:26.512288    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5762c366_b31f_488b_997b_f4ba1a053178.slice/crio-b1d4185d881ad8dfb265fa469c85587bd46d1ff48504698ceeed957eeece5d39 WatchSource:0}: Error finding container b1d4185d881ad8dfb265fa469c85587bd46d1ff48504698ceeed957eeece5d39: Status 404 returned error can't find the container with id b1d4185d881ad8dfb265fa469c85587bd46d1ff48504698ceeed957eeece5d39
Nov 03 13:29:26.515490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.515452    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:26.515736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.515533    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:26.515736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:26.515571    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:29:27.007991 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:27.007954    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:27.405315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:27.405194    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" event={"ID":"5762c366-b31f-488b-997b-f4ba1a053178","Type":"ContainerStarted","Data":"57e453dbe7760b554876c1ad31a01f5a693552f1809f0277b0ec0a566b9477ad"}
Nov 03 13:29:27.405315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:27.405290    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" event={"ID":"5762c366-b31f-488b-997b-f4ba1a053178","Type":"ContainerStarted","Data":"b1d4185d881ad8dfb265fa469c85587bd46d1ff48504698ceeed957eeece5d39"}
Nov 03 13:29:27.407121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:27.407094    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:27.408393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:27.408367    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" event={"ID":"2ae9c04b-23b3-4ea2-baf0-c8940a31044a","Type":"ContainerStarted","Data":"8f70d9b19a0988d125f9c4008c3897cfbbda84ec4aea04215df34cfcd7cf56b1"}
Nov 03 13:29:27.408497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:27.408422    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" event={"ID":"2ae9c04b-23b3-4ea2-baf0-c8940a31044a","Type":"ContainerStarted","Data":"3d706de228656c7686b8534c09bdd4d3c79c6b18d42dde66eb7ddd0ea427981c"}
Nov 03 13:29:27.474238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:27.453079    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podStartSLOduration=2.453056977 podStartE2EDuration="2.453056977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:27.449709897 +0000 UTC m=+1208.527474923" watchObservedRunningTime="2026-11-03 13:29:27.453056977 +0000 UTC m=+1208.530822005"
Nov 03 13:29:27.559240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:27.552173    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:28.075236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:28.073836    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:28.075236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:28.073951    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 13:29:28.075236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:28.073993    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 13:29:28.213469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:28.213435    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:28.215634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:28.215602    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 13:29:28.215900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:28.215664    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 13:29:28.564233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:28.563677    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:29.068270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:29.067847    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 13:29:29.068270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:29.067913    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 13:29:29.138235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:29.129581    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 13:29:29.138235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:29.129638    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 13:29:29.362909 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:29.361749    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:30.044232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:30.036732    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:30.078053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:30.078019    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 13:29:30.078367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:30.078322    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 13:29:30.132770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:30.132735    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 13:29:30.133086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:30.133042    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 13:29:30.739842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:30.739760    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:31.068297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:31.067629    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 13:29:31.068297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:31.067694    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 13:29:31.137357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:31.137321    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 13:29:31.137625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:31.137593    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 13:29:31.550415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:31.545631    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:32.067661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:32.067623    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 13:29:32.067973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:32.067939    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 13:29:32.131234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:32.130601    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 13:29:32.131234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:32.130672    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 13:29:32.204809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:32.204765    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:32.832080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:32.831779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:33.067952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:33.067907    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 13:29:33.068149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:33.067984    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 13:29:33.126318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:33.126227    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 13:29:33.126318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:33.126285    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 13:29:33.376736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:33.376652    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:33.979284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:33.979205    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:34.067869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:34.067834    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 13:29:34.068075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:34.067907    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 13:29:34.086446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:34.086373    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podStartSLOduration=9.086354467 podStartE2EDuration="9.086354467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:27.493592944 +0000 UTC m=+1208.571357969" watchObservedRunningTime="2026-11-03 13:29:34.086354467 +0000 UTC m=+1215.164119495"
Nov 03 13:29:34.087159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:34.087140    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"]
Nov 03 13:29:34.115261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:34.115231    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41b4d1-wzdkt"]
Nov 03 13:29:34.125673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:34.125644    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 13:29:34.125765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:34.125696    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 13:29:34.772649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:34.770440    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:35.095063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:35.094969    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc9bd4b-8f70-4dc2-b168-8dc60745d310" path="/var/lib/kubelet/pods/7dc9bd4b-8f70-4dc2-b168-8dc60745d310/volumes"
Nov 03 13:29:35.350732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:35.350643    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:35.530784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:35.530757    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:35.531129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:35.531106    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:35.532897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:35.532873    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk"
Nov 03 13:29:35.664452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:35.664373    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:35.664993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:35.664922    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:35.666639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:35.666610    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4"
Nov 03 13:29:36.009794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:36.009758    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:36.704036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:36.703992    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:37.452941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:37.452903    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:38.030438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:38.030404    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"]
Nov 03 13:29:38.035038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:38.035015    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc374748-mfqnk"]
Nov 03 13:29:38.202437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:38.202402    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:38.895608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:38.895575    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:39.086408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:39.086379    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90da7188-9990-498b-ad2e-71d477adbce9" path="/var/lib/kubelet/pods/90da7188-9990-498b-ad2e-71d477adbce9/volumes"
Nov 03 13:29:39.523606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:39.523574    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:40.144783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:40.144750    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:40.758115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:40.758083    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:40.899521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:40.899494    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:41.118552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:41.118474    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"]
Nov 03 13:29:41.301886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:41.301852    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:42.023822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:42.023782    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:42.892417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:42.892351    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:43.629641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:43.629568    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:44.026580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:44.026544    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"]
Nov 03 13:29:44.030968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:44.030944    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0055b7d-b9x8d"]
Nov 03 13:29:44.232440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:44.232396    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:44.797698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:44.796172    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:45.085651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:45.085578    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b9c406-e782-41c5-99c8-2887cf33bed3" path="/var/lib/kubelet/pods/72b9c406-e782-41c5-99c8-2887cf33bed3/volumes"
Nov 03 13:29:45.547099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:45.546838    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:46.119225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.119181    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg" podUID="bee30c13-51fe-46ef-a7ec-eebbd05f4644" containerName="payload-processing" containerID="cri-o://c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1" gracePeriod=25
Nov 03 13:29:46.362159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.362107    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:46.454997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.454963    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:29:46.535688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.535660    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-plugins-config-volume\") pod \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") "
Nov 03 13:29:46.535854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.535764    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-kube-api-access-rjjxb\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-kube-api-access-rjjxb\") pod \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") "
Nov 03 13:29:46.535854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.535795    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-provider-credentials\") pod \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") "
Nov 03 13:29:46.536033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.536014    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-routing-overlay-volume\") pod \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\" (UID: \"bee30c13-51fe-46ef-a7ec-eebbd05f4644\") "
Nov 03 13:29:46.536085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.536045    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-plugins-config-volume" pod "bee30c13-51fe-46ef-a7ec-eebbd05f4644" (UID: "bee30c13-51fe-46ef-a7ec-eebbd05f4644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:46.536384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.536362    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-routing-overlay-volume" pod "bee30c13-51fe-46ef-a7ec-eebbd05f4644" (UID: "bee30c13-51fe-46ef-a7ec-eebbd05f4644"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:46.536661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.536641    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:46.536726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.536664    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee30c13-51fe-46ef-a7ec-eebbd05f4644-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:46.538074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.538051    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-provider-credentials" pod "bee30c13-51fe-46ef-a7ec-eebbd05f4644" (UID: "bee30c13-51fe-46ef-a7ec-eebbd05f4644"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:46.538158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.538105    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-kube-api-access-rjjxb" pod "bee30c13-51fe-46ef-a7ec-eebbd05f4644" (UID: "bee30c13-51fe-46ef-a7ec-eebbd05f4644"). InnerVolumeSpecName "kube-api-access-rjjxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:46.638184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.638150    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjjxb\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-kube-api-access-rjjxb\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:46.638184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.638183    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bee30c13-51fe-46ef-a7ec-eebbd05f4644-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:46.708547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.708512    5420 generic.go:356] "Generic (PLEG): container finished" podID="bee30c13-51fe-46ef-a7ec-eebbd05f4644" containerID="c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1" exitCode=0
Nov 03 13:29:46.708737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.708562    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"
Nov 03 13:29:46.708737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.708597    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg" event={"ID":"bee30c13-51fe-46ef-a7ec-eebbd05f4644","Type":"ContainerDied","Data":"c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1"}
Nov 03 13:29:46.708737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.708652    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg" event={"ID":"bee30c13-51fe-46ef-a7ec-eebbd05f4644","Type":"ContainerDied","Data":"ea3626d038248ee375a3c970348a71dce15ce51523200c44bd25b18c137dba51"}
Nov 03 13:29:46.708737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.708680    5420 scope.go:122] "RemoveContainer" containerID="c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1"
Nov 03 13:29:46.725046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.724989    5420 scope.go:122] "RemoveContainer" containerID="c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1"
Nov 03 13:29:46.725558 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:29:46.725530    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1\": container with ID starting with c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1 not found: ID does not exist" containerID="c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1"
Nov 03 13:29:46.725640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.725568    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1"} err="failed to get container status \"c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1\": rpc error: code = NotFound desc = could not find container \"c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1\": container with ID starting with c360f0f5bea151be444f31b8849b0abd81b0e5b5bbe8bb94c352a113c95455b1 not found: ID does not exist"
Nov 03 13:29:46.739924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.739896    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"]
Nov 03 13:29:46.745600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:46.745577    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676c64gg"]
Nov 03 13:29:47.043438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:47.043323    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:47.086040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:47.086010    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee30c13-51fe-46ef-a7ec-eebbd05f4644" path="/var/lib/kubelet/pods/bee30c13-51fe-46ef-a7ec-eebbd05f4644/volumes"
Nov 03 13:29:47.641001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:47.640972    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:47.643325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:47.643297    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"]
Nov 03 13:29:47.643902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:47.643865    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy" containerID="cri-o://f301f0861de0d3848f92f8e05c6e465b513e09438c9a678dbac69ce8f36be18d" gracePeriod=30
Nov 03 13:29:47.689872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:47.689836    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"]
Nov 03 13:29:47.690390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:47.690361    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy" containerID="cri-o://2e16ab8aee9d12a55b986cb41472b3c866875baa1a7945669d4594fff9c49b08" gracePeriod=30
Nov 03 13:29:48.420907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:48.420871    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:49.015237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:49.013864    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:49.633158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:49.633124    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:50.230740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:50.230594    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:50.805050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:50.804948    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:51.471107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:51.471033    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:52.173716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:52.173671    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:52.857808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:52.857771    5420 generic.go:356] "Generic (PLEG): container finished" podID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerID="f301f0861de0d3848f92f8e05c6e465b513e09438c9a678dbac69ce8f36be18d" exitCode=0
Nov 03 13:29:52.858170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:52.857913    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" event={"ID":"adb963ee-4cb9-4f6a-945d-5fbbf05af5da","Type":"ContainerDied","Data":"f301f0861de0d3848f92f8e05c6e465b513e09438c9a678dbac69ce8f36be18d"}
Nov 03 13:29:52.860354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:52.860322    5420 generic.go:356] "Generic (PLEG): container finished" podID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerID="2e16ab8aee9d12a55b986cb41472b3c866875baa1a7945669d4594fff9c49b08" exitCode=0
Nov 03 13:29:52.860548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:52.860487    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" event={"ID":"3fd652cb-d578-4f95-b538-ed75795b5ddd","Type":"ContainerDied","Data":"2e16ab8aee9d12a55b986cb41472b3c866875baa1a7945669d4594fff9c49b08"}
Nov 03 13:29:52.920697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:52.920669    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:53.046977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.046946    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:29:53.075474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.075445    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:29:53.185712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.185681    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-envoy\") pod \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") "
Nov 03 13:29:53.185888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.185723    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd652cb-d578-4f95-b538-ed75795b5ddd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd652cb-d578-4f95-b538-ed75795b5ddd-istiod-ca-cert\") pod \"3fd652cb-d578-4f95-b538-ed75795b5ddd\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") "
Nov 03 13:29:53.185888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.185765    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-data\") pod \"3fd652cb-d578-4f95-b538-ed75795b5ddd\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") "
Nov 03 13:29:53.185888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.185800    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-certs\") pod \"3fd652cb-d578-4f95-b538-ed75795b5ddd\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") "
Nov 03 13:29:53.185888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.185838    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-credential-socket\") pod \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") "
Nov 03 13:29:53.186118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.185903    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-kube-api-access-ndhkp\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-kube-api-access-ndhkp\") pod \"3fd652cb-d578-4f95-b538-ed75795b5ddd\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") "
Nov 03 13:29:53.186118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.185955    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istiod-ca-cert\") pod \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") "
Nov 03 13:29:53.186118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.185986    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-envoy\") pod \"3fd652cb-d578-4f95-b538-ed75795b5ddd\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") "
Nov 03 13:29:53.186118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186017    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-socket\") pod \"3fd652cb-d578-4f95-b538-ed75795b5ddd\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") "
Nov 03 13:29:53.186118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186087    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-data" pod "3fd652cb-d578-4f95-b538-ed75795b5ddd" (UID: "3fd652cb-d578-4f95-b538-ed75795b5ddd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.186435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186120    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-data\") pod \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") "
Nov 03 13:29:53.186435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186129    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-certs" pod "3fd652cb-d578-4f95-b538-ed75795b5ddd" (UID: "3fd652cb-d578-4f95-b538-ed75795b5ddd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.186435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186150    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-credential-socket" pod "adb963ee-4cb9-4f6a-945d-5fbbf05af5da" (UID: "adb963ee-4cb9-4f6a-945d-5fbbf05af5da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.186435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186189    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-podinfo\") pod \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") "
Nov 03 13:29:53.186435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186283    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-token\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-token\") pod \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") "
Nov 03 13:29:53.186435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186333    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-credential-socket\") pod \"3fd652cb-d578-4f95-b538-ed75795b5ddd\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") "
Nov 03 13:29:53.186435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186354    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd652cb-d578-4f95-b538-ed75795b5ddd-istiod-ca-cert" pod "3fd652cb-d578-4f95-b538-ed75795b5ddd" (UID: "3fd652cb-d578-4f95-b538-ed75795b5ddd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:53.186435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186371    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-socket\") pod \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") "
Nov 03 13:29:53.186864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186411    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-data" pod "adb963ee-4cb9-4f6a-945d-5fbbf05af5da" (UID: "adb963ee-4cb9-4f6a-945d-5fbbf05af5da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.186864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186466    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-socket" pod "3fd652cb-d578-4f95-b538-ed75795b5ddd" (UID: "3fd652cb-d578-4f95-b538-ed75795b5ddd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.186864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186568    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-kube-api-access-nsgpp\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-kube-api-access-nsgpp\") pod \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") "
Nov 03 13:29:53.186864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186571    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istiod-ca-cert" pod "adb963ee-4cb9-4f6a-945d-5fbbf05af5da" (UID: "adb963ee-4cb9-4f6a-945d-5fbbf05af5da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:29:53.186864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186579    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-socket" pod "adb963ee-4cb9-4f6a-945d-5fbbf05af5da" (UID: "adb963ee-4cb9-4f6a-945d-5fbbf05af5da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.186864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186731    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-credential-socket" pod "3fd652cb-d578-4f95-b538-ed75795b5ddd" (UID: "3fd652cb-d578-4f95-b538-ed75795b5ddd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.186864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186764    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-podinfo\") pod \"3fd652cb-d578-4f95-b538-ed75795b5ddd\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") "
Nov 03 13:29:53.186864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186802    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-token\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-token\") pod \"3fd652cb-d578-4f95-b538-ed75795b5ddd\" (UID: \"3fd652cb-d578-4f95-b538-ed75795b5ddd\") "
Nov 03 13:29:53.187319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.186892    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-certs\") pod \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\" (UID: \"adb963ee-4cb9-4f6a-945d-5fbbf05af5da\") "
Nov 03 13:29:53.187872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.187844    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.188035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.187876    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.188035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.187901    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd652cb-d578-4f95-b538-ed75795b5ddd-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.188035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.187916    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.188035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.187930    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.188035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.187945    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.188035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.187961    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.188035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.187975    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.188035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.187991    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.188827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.188524    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-envoy" pod "adb963ee-4cb9-4f6a-945d-5fbbf05af5da" (UID: "adb963ee-4cb9-4f6a-945d-5fbbf05af5da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.188950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.188894    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-certs" pod "adb963ee-4cb9-4f6a-945d-5fbbf05af5da" (UID: "adb963ee-4cb9-4f6a-945d-5fbbf05af5da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.189040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.188996    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-envoy" pod "3fd652cb-d578-4f95-b538-ed75795b5ddd" (UID: "3fd652cb-d578-4f95-b538-ed75795b5ddd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:29:53.189318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.189289    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-podinfo" pod "adb963ee-4cb9-4f6a-945d-5fbbf05af5da" (UID: "adb963ee-4cb9-4f6a-945d-5fbbf05af5da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:29:53.189318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.189296    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-kube-api-access-nsgpp" pod "adb963ee-4cb9-4f6a-945d-5fbbf05af5da" (UID: "adb963ee-4cb9-4f6a-945d-5fbbf05af5da"). InnerVolumeSpecName "kube-api-access-nsgpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:53.189552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.189535    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-token" pod "adb963ee-4cb9-4f6a-945d-5fbbf05af5da" (UID: "adb963ee-4cb9-4f6a-945d-5fbbf05af5da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:53.189624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.189605    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-podinfo" pod "3fd652cb-d578-4f95-b538-ed75795b5ddd" (UID: "3fd652cb-d578-4f95-b538-ed75795b5ddd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:29:53.190304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.190281    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-kube-api-access-ndhkp" pod "3fd652cb-d578-4f95-b538-ed75795b5ddd" (UID: "3fd652cb-d578-4f95-b538-ed75795b5ddd"). InnerVolumeSpecName "kube-api-access-ndhkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:53.190388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.190380    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-token" pod "3fd652cb-d578-4f95-b538-ed75795b5ddd" (UID: "3fd652cb-d578-4f95-b538-ed75795b5ddd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:29:53.289887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.289813    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.289887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.289841    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndhkp\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-kube-api-access-ndhkp\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.289887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.289851    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.289887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.289861    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.289887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.289870    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.289887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.289880    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsgpp\" (UniqueName: \"kubernetes.io/projected/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-kube-api-access-nsgpp\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.289887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.289890    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.290238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.289900    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd652cb-d578-4f95-b538-ed75795b5ddd-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.290238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.289908    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb963ee-4cb9-4f6a-945d-5fbbf05af5da-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:29:53.607464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.607374    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:53.871712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.871627    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"
Nov 03 13:29:53.871712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.871658    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w" event={"ID":"adb963ee-4cb9-4f6a-945d-5fbbf05af5da","Type":"ContainerDied","Data":"d9c57d3c0e09d5441753f7c6602d409dba9de52cc70f42454b7c07ffabf3c635"}
Nov 03 13:29:53.872272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.871737    5420 scope.go:122] "RemoveContainer" containerID="f301f0861de0d3848f92f8e05c6e465b513e09438c9a678dbac69ce8f36be18d"
Nov 03 13:29:53.873521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.873500    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"
Nov 03 13:29:53.873630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.873531    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x" event={"ID":"3fd652cb-d578-4f95-b538-ed75795b5ddd","Type":"ContainerDied","Data":"1dcb92e2f68e2fa4a873bcbd624a0082ddeb75a47edc74aeeb79eb00b8e2a164"}
Nov 03 13:29:53.885458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.885423    5420 scope.go:122] "RemoveContainer" containerID="2e16ab8aee9d12a55b986cb41472b3c866875baa1a7945669d4594fff9c49b08"
Nov 03 13:29:53.901096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.901064    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"]
Nov 03 13:29:53.905642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.905619    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-8m68w"]
Nov 03 13:29:53.918739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.918717    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"]
Nov 03 13:29:53.922899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:53.922879    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-b662x"]
Nov 03 13:29:54.222658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:54.222607    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:54.689859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:54.689787    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:29:54.732262 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:29:54.732223    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:29:54.832844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:54.832502    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:55.095340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:55.095308    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" path="/var/lib/kubelet/pods/3fd652cb-d578-4f95-b538-ed75795b5ddd/volumes"
Nov 03 13:29:55.095842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:55.095827    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" path="/var/lib/kubelet/pods/adb963ee-4cb9-4f6a-945d-5fbbf05af5da/volumes"
Nov 03 13:29:55.946162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:55.943752    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:56.674238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:56.671684    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:57.406448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:57.406409    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:58.470766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:58.470680    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:59.444426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:29:59.444390    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:00.038118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.038081    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:00.151534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.151503    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"]
Nov 03 13:30:00.152656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.152634    5420 cpu_manager.go:414] "containerMap: removing container" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy"
Nov 03 13:30:00.152750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.152659    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy"
Nov 03 13:30:00.152750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.152681    5420 cpu_manager.go:414] "containerMap: removing container" podUID="bee30c13-51fe-46ef-a7ec-eebbd05f4644" containerName="payload-processing"
Nov 03 13:30:00.152750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.152689    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee30c13-51fe-46ef-a7ec-eebbd05f4644" containerName="payload-processing"
Nov 03 13:30:00.152750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.152723    5420 cpu_manager.go:414] "containerMap: removing container" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy"
Nov 03 13:30:00.152750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.152731    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy"
Nov 03 13:30:00.153001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.152904    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd652cb-d578-4f95-b538-ed75795b5ddd" containerName="istio-proxy"
Nov 03 13:30:00.153001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.152932    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb963ee-4cb9-4f6a-945d-5fbbf05af5da" containerName="istio-proxy"
Nov 03 13:30:00.153001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.152941    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee30c13-51fe-46ef-a7ec-eebbd05f4644" containerName="payload-processing"
Nov 03 13:30:00.158760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.158740    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:00.169690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.169667    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"]
Nov 03 13:30:00.174442 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.174418    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:30:00.174553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.174454    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:00.174553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.174463    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"]
Nov 03 13:30:00.174812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.174793    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:30:00.174901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.174880    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:30:00.179456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.179433    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:00.182548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.182526    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"]
Nov 03 13:30:00.188285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.188266    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:00.205084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.195307    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"]
Nov 03 13:30:00.208085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.208063    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"]
Nov 03 13:30:00.208203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.208105    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"]
Nov 03 13:30:00.208203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.208174    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:00.209797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.209776    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"]
Nov 03 13:30:00.215258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.215235    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"]
Nov 03 13:30:00.221814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.221793    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"]
Nov 03 13:30:00.235454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.235430    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:00.236285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.236191    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"]
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244054    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkvx6\" (UniqueName: \"kubernetes.io/projected/c6624d06-66d9-40b9-8e96-7f9018f66bce-kube-api-access-rkvx6\") pod \"maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8\" (UID: \"c6624d06-66d9-40b9-8e96-7f9018f66bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244118    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f2f37b-476c-4ddb-b013-dfc7d187a74c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl\" (UID: \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244153    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mtc\" (UniqueName: \"kubernetes.io/projected/55f2f37b-476c-4ddb-b013-dfc7d187a74c-kube-api-access-p7mtc\") pod \"maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl\" (UID: \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244200    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzbk9\" (UniqueName: \"kubernetes.io/projected/f5ceef64-17be-4b59-a562-da452c10930b-kube-api-access-qzbk9\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct\" (UID: \"f5ceef64-17be-4b59-a562-da452c10930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244410    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzncw\" (UniqueName: \"kubernetes.io/projected/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-kube-api-access-pzncw\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw\" (UID: \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244476    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdh24\" (UniqueName: \"kubernetes.io/projected/2c21bb75-b140-4359-af93-f4234cdae01d-kube-api-access-cdh24\") pod \"maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh\" (UID: \"2c21bb75-b140-4359-af93-f4234cdae01d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244542    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6624d06-66d9-40b9-8e96-7f9018f66bce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8\" (UID: \"c6624d06-66d9-40b9-8e96-7f9018f66bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244573    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw\" (UID: \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244631    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21bb75-b140-4359-af93-f4234cdae01d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh\" (UID: \"2c21bb75-b140-4359-af93-f4234cdae01d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:00.244886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.244726    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceef64-17be-4b59-a562-da452c10930b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct\" (UID: \"f5ceef64-17be-4b59-a562-da452c10930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:00.256561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.256544    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"]
Nov 03 13:30:00.256745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.256730    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"]
Nov 03 13:30:00.256896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.256883    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"]
Nov 03 13:30:00.257120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.257106    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:00.262822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.258945    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"]
Nov 03 13:30:00.265907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.265510    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:00.277020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.276999    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"]
Nov 03 13:30:00.284355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.284334    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:00.285281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.285264    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"]
Nov 03 13:30:00.293144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.293098    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"]
Nov 03 13:30:00.339474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.339446    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"]
Nov 03 13:30:00.344368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.344346    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.345809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.345785    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-kube-api-access-c4zbj\") pod \"maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk\" (UID: \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:00.345917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.345831    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceef64-17be-4b59-a562-da452c10930b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct\" (UID: \"f5ceef64-17be-4b59-a562-da452c10930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:00.345917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.345903    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkvx6\" (UniqueName: \"kubernetes.io/projected/c6624d06-66d9-40b9-8e96-7f9018f66bce-kube-api-access-rkvx6\") pod \"maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8\" (UID: \"c6624d06-66d9-40b9-8e96-7f9018f66bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:00.346029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.345944    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c6bc18-88af-45d7-a76d-d5a1e683cdba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7\" (UID: \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:00.346029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.345978    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f2f37b-476c-4ddb-b013-dfc7d187a74c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl\" (UID: \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:00.346029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346006    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mtc\" (UniqueName: \"kubernetes.io/projected/55f2f37b-476c-4ddb-b013-dfc7d187a74c-kube-api-access-p7mtc\") pod \"maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl\" (UID: \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:00.346184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346042    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzbk9\" (UniqueName: \"kubernetes.io/projected/f5ceef64-17be-4b59-a562-da452c10930b-kube-api-access-qzbk9\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct\" (UID: \"f5ceef64-17be-4b59-a562-da452c10930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:00.346184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346121    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627\" (UID: \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:00.346184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346141    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzncw\" (UniqueName: \"kubernetes.io/projected/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-kube-api-access-pzncw\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw\" (UID: \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:00.346184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346170    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg\" (UID: \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:00.346409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346241    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdh24\" (UniqueName: \"kubernetes.io/projected/2c21bb75-b140-4359-af93-f4234cdae01d-kube-api-access-cdh24\") pod \"maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh\" (UID: \"2c21bb75-b140-4359-af93-f4234cdae01d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:00.346409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346273    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnfsb\" (UniqueName: \"kubernetes.io/projected/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-kube-api-access-cnfsb\") pod \"maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627\" (UID: \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:00.346409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346290    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk\" (UID: \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:00.346409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346314    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wg2\" (UniqueName: \"kubernetes.io/projected/20c6bc18-88af-45d7-a76d-d5a1e683cdba-kube-api-access-h6wg2\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7\" (UID: \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:00.346409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346339    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2857k\" (UniqueName: \"kubernetes.io/projected/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-kube-api-access-2857k\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg\" (UID: \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:00.346409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346371    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6624d06-66d9-40b9-8e96-7f9018f66bce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8\" (UID: \"c6624d06-66d9-40b9-8e96-7f9018f66bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:00.346409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346398    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw\" (UID: \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:00.346776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346676    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceef64-17be-4b59-a562-da452c10930b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct\" (UID: \"f5ceef64-17be-4b59-a562-da452c10930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:00.346961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.346941    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6624d06-66d9-40b9-8e96-7f9018f66bce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8\" (UID: \"c6624d06-66d9-40b9-8e96-7f9018f66bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:00.347083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.347070    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21bb75-b140-4359-af93-f4234cdae01d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh\" (UID: \"2c21bb75-b140-4359-af93-f4234cdae01d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:00.347392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.347374    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw\" (UID: \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:00.347592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.347570    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f2f37b-476c-4ddb-b013-dfc7d187a74c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl\" (UID: \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:00.347668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.347653    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21bb75-b140-4359-af93-f4234cdae01d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh\" (UID: \"2c21bb75-b140-4359-af93-f4234cdae01d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:00.361450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.361423    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"]
Nov 03 13:30:00.364353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.364018    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzncw\" (UniqueName: \"kubernetes.io/projected/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-kube-api-access-pzncw\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw\" (UID: \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:00.364353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.364306    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mtc\" (UniqueName: \"kubernetes.io/projected/55f2f37b-476c-4ddb-b013-dfc7d187a74c-kube-api-access-p7mtc\") pod \"maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl\" (UID: \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:00.369340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.369320    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzbk9\" (UniqueName: \"kubernetes.io/projected/f5ceef64-17be-4b59-a562-da452c10930b-kube-api-access-qzbk9\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct\" (UID: \"f5ceef64-17be-4b59-a562-da452c10930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:00.373659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.373637    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkvx6\" (UniqueName: \"kubernetes.io/projected/c6624d06-66d9-40b9-8e96-7f9018f66bce-kube-api-access-rkvx6\") pod \"maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8\" (UID: \"c6624d06-66d9-40b9-8e96-7f9018f66bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:00.373837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.373813    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdh24\" (UniqueName: \"kubernetes.io/projected/2c21bb75-b140-4359-af93-f4234cdae01d-kube-api-access-cdh24\") pod \"maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh\" (UID: \"2c21bb75-b140-4359-af93-f4234cdae01d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:00.375687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.375671    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-1a18c8-openshift-default-dockercfg-ff4vz\""
Nov 03 13:30:00.448335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448306    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2857k\" (UniqueName: \"kubernetes.io/projected/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-kube-api-access-2857k\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg\" (UID: \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:00.448469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448373    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-token\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.448469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448399    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.448469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448428    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-certs\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.448469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448446    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-podinfo\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.448689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448581    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtml\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-kube-api-access-mhtml\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.448689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448609    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-kube-api-access-c4zbj\") pod \"maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk\" (UID: \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:00.448689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448648    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c6bc18-88af-45d7-a76d-d5a1e683cdba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7\" (UID: \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:00.448851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448779    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istiod-ca-cert\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.448851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448819    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627\" (UID: \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:00.448961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448865    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg\" (UID: \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:00.448961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448892    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-envoy\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.448961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448940    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-data\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.449138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.448996    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-credential-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.449138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.449025    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnfsb\" (UniqueName: \"kubernetes.io/projected/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-kube-api-access-cnfsb\") pod \"maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627\" (UID: \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:00.449138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.449049    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk\" (UID: \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:00.449138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.449068    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wg2\" (UniqueName: \"kubernetes.io/projected/20c6bc18-88af-45d7-a76d-d5a1e683cdba-kube-api-access-h6wg2\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7\" (UID: \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:00.449416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.449324    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c6bc18-88af-45d7-a76d-d5a1e683cdba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7\" (UID: \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:00.449550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.449531    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627\" (UID: \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:00.449640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.449621    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg\" (UID: \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:00.449731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.449694    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk\" (UID: \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:00.465175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.465152    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2857k\" (UniqueName: \"kubernetes.io/projected/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-kube-api-access-2857k\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg\" (UID: \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:00.467078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.467054    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wg2\" (UniqueName: \"kubernetes.io/projected/20c6bc18-88af-45d7-a76d-d5a1e683cdba-kube-api-access-h6wg2\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7\" (UID: \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:00.467305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.467285    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-kube-api-access-c4zbj\") pod \"maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk\" (UID: \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:00.467659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.467642    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnfsb\" (UniqueName: \"kubernetes.io/projected/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-kube-api-access-cnfsb\") pod \"maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627\" (UID: \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:00.477120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.477103    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:00.515136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.515109    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:00.538117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.538090    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.570856    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istiod-ca-cert\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.570935    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-envoy\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.570992    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-data\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.571037    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-credential-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.571127    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-token\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.571159    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.571203    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-certs\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.571247    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-podinfo\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.571343    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtml\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-kube-api-access-mhtml\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.572422    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istiod-ca-cert\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.572917    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.573256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.573103    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:00.574021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.573422    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:00.574021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.573493    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-certs\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.574021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.573899    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-data\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.574021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.573947    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-credential-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.575735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.575677    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-envoy\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.577478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.577428    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-podinfo\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.581721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.581681    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-token\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.582405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.582382    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtml\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-kube-api-access-mhtml\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.586255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.586194    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:00.599979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.599403    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:00.632847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.631930    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:00.638964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.638878    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:00.684701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.684488    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:00.776572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.776524    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"]
Nov 03 13:30:00.811377 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:30:00.809618    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6624d06_66d9_40b9_8e96_7f9018f66bce.slice/crio-7688d9d3c84bd54f1a380bb05d52793d794255f2d2bc062324a7ea4624c550b9 WatchSource:0}: Error finding container 7688d9d3c84bd54f1a380bb05d52793d794255f2d2bc062324a7ea4624c550b9: Status 404 returned error can't find the container with id 7688d9d3c84bd54f1a380bb05d52793d794255f2d2bc062324a7ea4624c550b9
Nov 03 13:30:00.957768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:00.957738    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" event={"ID":"c6624d06-66d9-40b9-8e96-7f9018f66bce","Type":"ContainerStarted","Data":"7688d9d3c84bd54f1a380bb05d52793d794255f2d2bc062324a7ea4624c550b9"}
Nov 03 13:30:01.062431 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:30:01.062405    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-5fcafeea87dad8f1bc1a573c4d0accafc242adbf1b1fe6b71148abc70fe20afc WatchSource:0}: Error finding container 5fcafeea87dad8f1bc1a573c4d0accafc242adbf1b1fe6b71148abc70fe20afc: Status 404 returned error can't find the container with id 5fcafeea87dad8f1bc1a573c4d0accafc242adbf1b1fe6b71148abc70fe20afc
Nov 03 13:30:01.064615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.064591    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"]
Nov 03 13:30:01.107615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.101625    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:01.121734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.121707    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"]
Nov 03 13:30:01.122341 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:30:01.122310    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aaea81a_1f45_4262_94a5_ae0b5bf486c1.slice/crio-e60aa7d902a0824caadf379a10ba6878797ec63c9ef79dd8ceb199eb198366ff WatchSource:0}: Error finding container e60aa7d902a0824caadf379a10ba6878797ec63c9ef79dd8ceb199eb198366ff: Status 404 returned error can't find the container with id e60aa7d902a0824caadf379a10ba6878797ec63c9ef79dd8ceb199eb198366ff
Nov 03 13:30:01.191572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.191549    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"]
Nov 03 13:30:01.196407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.196386    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"]
Nov 03 13:30:01.223450 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:30:01.223421    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c21bb75_b140_4359_af93_f4234cdae01d.slice/crio-b5f0c9dd10c1ec0b7574af03751898ffc52096faa89f5a2c939426bcebdd14e1 WatchSource:0}: Error finding container b5f0c9dd10c1ec0b7574af03751898ffc52096faa89f5a2c939426bcebdd14e1: Status 404 returned error can't find the container with id b5f0c9dd10c1ec0b7574af03751898ffc52096faa89f5a2c939426bcebdd14e1
Nov 03 13:30:01.225024 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:30:01.225000    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f2f37b_476c_4ddb_b013_dfc7d187a74c.slice/crio-c8db89fd8bbc4a8c0c737ec1f09fa570ab804c7ba522d38b13fb64f372056991 WatchSource:0}: Error finding container c8db89fd8bbc4a8c0c737ec1f09fa570ab804c7ba522d38b13fb64f372056991: Status 404 returned error can't find the container with id c8db89fd8bbc4a8c0c737ec1f09fa570ab804c7ba522d38b13fb64f372056991
Nov 03 13:30:01.242786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.242759    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"]
Nov 03 13:30:01.249582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.249561    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"]
Nov 03 13:30:01.253001 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:30:01.252965    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe356c0_c8f6_4b0e_b9fe_9a1946742d00.slice/crio-9312f87e5d930fc17d7ce79ce045a21b849e4a83bd7629de1456b9f817eeeeeb WatchSource:0}: Error finding container 9312f87e5d930fc17d7ce79ce045a21b849e4a83bd7629de1456b9f817eeeeeb: Status 404 returned error can't find the container with id 9312f87e5d930fc17d7ce79ce045a21b849e4a83bd7629de1456b9f817eeeeeb
Nov 03 13:30:01.260000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.259980    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"]
Nov 03 13:30:01.266485 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:30:01.266455    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c6bc18_88af_45d7_a76d_d5a1e683cdba.slice/crio-799ea4e998b5cd3b27697732f3d7005018f758162c92b1afc8420d5cc80b97f5 WatchSource:0}: Error finding container 799ea4e998b5cd3b27697732f3d7005018f758162c92b1afc8420d5cc80b97f5: Status 404 returned error can't find the container with id 799ea4e998b5cd3b27697732f3d7005018f758162c92b1afc8420d5cc80b97f5
Nov 03 13:30:01.276747 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:30:01.276470    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc48ac48_5efb_41c5_971f_a74cd1c3ac23.slice/crio-b980545192a7a36f514a1f58782a338343e232a4735cbd921fbbb2807a720e48 WatchSource:0}: Error finding container b980545192a7a36f514a1f58782a338343e232a4735cbd921fbbb2807a720e48: Status 404 returned error can't find the container with id b980545192a7a36f514a1f58782a338343e232a4735cbd921fbbb2807a720e48
Nov 03 13:30:01.283920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.283895    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"]
Nov 03 13:30:01.290296 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:30:01.286983    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6550bebc_8bb9_4d87_88cc_e4ad7c2f17cd.slice/crio-3fe9a134f3db98118ec00778f94b3ff1e735414993d7e0daeaa58bb8c806acc4 WatchSource:0}: Error finding container 3fe9a134f3db98118ec00778f94b3ff1e735414993d7e0daeaa58bb8c806acc4: Status 404 returned error can't find the container with id 3fe9a134f3db98118ec00778f94b3ff1e735414993d7e0daeaa58bb8c806acc4
Nov 03 13:30:01.290296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.288413    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:30:01.290296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.288467    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:30:01.290296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.288500    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:30:01.291295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.291270    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"]
Nov 03 13:30:01.994488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.994391    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:01.995263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.995235    5420 generic.go:356] "Generic (PLEG): container finished" podID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerID="b1add12d97d0978019adc608f946af01b726a00e68a54ba53a82a459274554e7" exitCode=60
Nov 03 13:30:01.995405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.995363    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" event={"ID":"c6624d06-66d9-40b9-8e96-7f9018f66bce","Type":"ContainerDied","Data":"b1add12d97d0978019adc608f946af01b726a00e68a54ba53a82a459274554e7"}
Nov 03 13:30:01.995877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:01.995857    5420 scope.go:122] "RemoveContainer" containerID="b1add12d97d0978019adc608f946af01b726a00e68a54ba53a82a459274554e7"
Nov 03 13:30:02.003400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.003373    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" event={"ID":"2aaea81a-1f45-4262-94a5-ae0b5bf486c1","Type":"ContainerStarted","Data":"aabf8f4dfb17fc6e5e3398c8c794842df3b5113d76e7967146a1cec63fec1024"}
Nov 03 13:30:02.003508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.003433    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" event={"ID":"2aaea81a-1f45-4262-94a5-ae0b5bf486c1","Type":"ContainerStarted","Data":"e60aa7d902a0824caadf379a10ba6878797ec63c9ef79dd8ceb199eb198366ff"}
Nov 03 13:30:02.005235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.005176    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" event={"ID":"20c6bc18-88af-45d7-a76d-d5a1e683cdba","Type":"ContainerStarted","Data":"799ea4e998b5cd3b27697732f3d7005018f758162c92b1afc8420d5cc80b97f5"}
Nov 03 13:30:02.007189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.007171    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" event={"ID":"cc48ac48-5efb-41c5-971f-a74cd1c3ac23","Type":"ContainerStarted","Data":"2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611"}
Nov 03 13:30:02.007300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.007225    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" event={"ID":"cc48ac48-5efb-41c5-971f-a74cd1c3ac23","Type":"ContainerStarted","Data":"b980545192a7a36f514a1f58782a338343e232a4735cbd921fbbb2807a720e48"}
Nov 03 13:30:02.009378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.009351    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" event={"ID":"55f2f37b-476c-4ddb-b013-dfc7d187a74c","Type":"ContainerStarted","Data":"b14e2efe5b0dcb3356d6269afd29acff8d261f7b39971982a42f820ccbfcd9eb"}
Nov 03 13:30:02.009617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.009403    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" event={"ID":"55f2f37b-476c-4ddb-b013-dfc7d187a74c","Type":"ContainerStarted","Data":"c8db89fd8bbc4a8c0c737ec1f09fa570ab804c7ba522d38b13fb64f372056991"}
Nov 03 13:30:02.011230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.011190    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" event={"ID":"2c21bb75-b140-4359-af93-f4234cdae01d","Type":"ContainerStarted","Data":"b5f0c9dd10c1ec0b7574af03751898ffc52096faa89f5a2c939426bcebdd14e1"}
Nov 03 13:30:02.012780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.012721    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" event={"ID":"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00","Type":"ContainerStarted","Data":"9312f87e5d930fc17d7ce79ce045a21b849e4a83bd7629de1456b9f817eeeeeb"}
Nov 03 13:30:02.018074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.018048    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" event={"ID":"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd","Type":"ContainerStarted","Data":"3fe9a134f3db98118ec00778f94b3ff1e735414993d7e0daeaa58bb8c806acc4"}
Nov 03 13:30:02.019938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.019914    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" event={"ID":"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8","Type":"ContainerStarted","Data":"846f67339ef583f844dc01f98d6ef760e5a5d5e8e5b6f9edba15f177246bee4b"}
Nov 03 13:30:02.022230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.022144    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" event={"ID":"f5ceef64-17be-4b59-a562-da452c10930b","Type":"ContainerStarted","Data":"fcdd3f4f0cab7ce8b30c84c24d33385d336fd58147f42d2e1ebaa6bdc5c1d42a"}
Nov 03 13:30:02.022230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.022185    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" event={"ID":"f5ceef64-17be-4b59-a562-da452c10930b","Type":"ContainerStarted","Data":"5fcafeea87dad8f1bc1a573c4d0accafc242adbf1b1fe6b71148abc70fe20afc"}
Nov 03 13:30:02.052924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.052862    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" podStartSLOduration=2.052843645 podStartE2EDuration="2.052843645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:02.048901968 +0000 UTC m=+1243.126666994" watchObservedRunningTime="2026-11-03 13:30:02.052843645 +0000 UTC m=+1243.130608685"
Nov 03 13:30:02.077362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.077284    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" podStartSLOduration=2.077265144 podStartE2EDuration="2.077265144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:02.070549709 +0000 UTC m=+1243.148314732" watchObservedRunningTime="2026-11-03 13:30:02.077265144 +0000 UTC m=+1243.155030190"
Nov 03 13:30:02.094092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.094042    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" podStartSLOduration=2.094024649 podStartE2EDuration="2.094024649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:02.089057847 +0000 UTC m=+1243.166822874" watchObservedRunningTime="2026-11-03 13:30:02.094024649 +0000 UTC m=+1243.171789678"
Nov 03 13:30:02.114989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.114942    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" podStartSLOduration=2.114929339 podStartE2EDuration="2.114929339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:02.107769611 +0000 UTC m=+1243.185534636" watchObservedRunningTime="2026-11-03 13:30:02.114929339 +0000 UTC m=+1243.192694358"
Nov 03 13:30:02.686068 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.686045    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:02.687470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.687448    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 13:30:02.687548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.687488    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 13:30:02.854234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:02.837030    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:03.054735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.054706    5420 generic.go:356] "Generic (PLEG): container finished" podID="f5ceef64-17be-4b59-a562-da452c10930b" containerID="fcdd3f4f0cab7ce8b30c84c24d33385d336fd58147f42d2e1ebaa6bdc5c1d42a" exitCode=60
Nov 03 13:30:03.054913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.054788    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" event={"ID":"f5ceef64-17be-4b59-a562-da452c10930b","Type":"ContainerDied","Data":"fcdd3f4f0cab7ce8b30c84c24d33385d336fd58147f42d2e1ebaa6bdc5c1d42a"}
Nov 03 13:30:03.055401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.055132    5420 scope.go:122] "RemoveContainer" containerID="fcdd3f4f0cab7ce8b30c84c24d33385d336fd58147f42d2e1ebaa6bdc5c1d42a"
Nov 03 13:30:03.057191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.057044    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" event={"ID":"c6624d06-66d9-40b9-8e96-7f9018f66bce","Type":"ContainerStarted","Data":"98f6b60fad2d27b960771c4d76c73adc38deb6e07687c457abae9db1ec6e14f7"}
Nov 03 13:30:03.058962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.058940    5420 generic.go:356] "Generic (PLEG): container finished" podID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerID="aabf8f4dfb17fc6e5e3398c8c794842df3b5113d76e7967146a1cec63fec1024" exitCode=60
Nov 03 13:30:03.059043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.059011    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" event={"ID":"2aaea81a-1f45-4262-94a5-ae0b5bf486c1","Type":"ContainerDied","Data":"aabf8f4dfb17fc6e5e3398c8c794842df3b5113d76e7967146a1cec63fec1024"}
Nov 03 13:30:03.059223 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.059187    5420 scope.go:122] "RemoveContainer" containerID="aabf8f4dfb17fc6e5e3398c8c794842df3b5113d76e7967146a1cec63fec1024"
Nov 03 13:30:03.061082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.061060    5420 generic.go:356] "Generic (PLEG): container finished" podID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerID="74777ea2522b002cf206101217bf09448412dd9abfca4f2c5a4201839983db95" exitCode=60
Nov 03 13:30:03.061171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.061151    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" event={"ID":"20c6bc18-88af-45d7-a76d-d5a1e683cdba","Type":"ContainerDied","Data":"74777ea2522b002cf206101217bf09448412dd9abfca4f2c5a4201839983db95"}
Nov 03 13:30:03.061356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.061340    5420 scope.go:122] "RemoveContainer" containerID="74777ea2522b002cf206101217bf09448412dd9abfca4f2c5a4201839983db95"
Nov 03 13:30:03.062839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.062820    5420 generic.go:356] "Generic (PLEG): container finished" podID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerID="b14e2efe5b0dcb3356d6269afd29acff8d261f7b39971982a42f820ccbfcd9eb" exitCode=60
Nov 03 13:30:03.062933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.062908    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" event={"ID":"55f2f37b-476c-4ddb-b013-dfc7d187a74c","Type":"ContainerDied","Data":"b14e2efe5b0dcb3356d6269afd29acff8d261f7b39971982a42f820ccbfcd9eb"}
Nov 03 13:30:03.063165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.063150    5420 scope.go:122] "RemoveContainer" containerID="b14e2efe5b0dcb3356d6269afd29acff8d261f7b39971982a42f820ccbfcd9eb"
Nov 03 13:30:03.064661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.064643    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c21bb75-b140-4359-af93-f4234cdae01d" containerID="3ab6377aefb3b5576d7f1c98cc97ef5061995bf05f77f1d44f4ac91b5478cd6c" exitCode=60
Nov 03 13:30:03.064738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.064723    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" event={"ID":"2c21bb75-b140-4359-af93-f4234cdae01d","Type":"ContainerDied","Data":"3ab6377aefb3b5576d7f1c98cc97ef5061995bf05f77f1d44f4ac91b5478cd6c"}
Nov 03 13:30:03.064931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.064916    5420 scope.go:122] "RemoveContainer" containerID="3ab6377aefb3b5576d7f1c98cc97ef5061995bf05f77f1d44f4ac91b5478cd6c"
Nov 03 13:30:03.066513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.066483    5420 generic.go:356] "Generic (PLEG): container finished" podID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerID="748ccf4f90e4313ef7f6c867adbeae3a8b82b86b9856ba218497438e6808da3a" exitCode=60
Nov 03 13:30:03.066615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.066563    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" event={"ID":"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00","Type":"ContainerDied","Data":"748ccf4f90e4313ef7f6c867adbeae3a8b82b86b9856ba218497438e6808da3a"}
Nov 03 13:30:03.066790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.066771    5420 scope.go:122] "RemoveContainer" containerID="748ccf4f90e4313ef7f6c867adbeae3a8b82b86b9856ba218497438e6808da3a"
Nov 03 13:30:03.069443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.069403    5420 generic.go:356] "Generic (PLEG): container finished" podID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerID="027f23361f5649e44ccbd04150d852acbbfa2dd1b52e9d37dd3261d97b6ab7da" exitCode=60
Nov 03 13:30:03.069565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.069483    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" event={"ID":"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd","Type":"ContainerDied","Data":"027f23361f5649e44ccbd04150d852acbbfa2dd1b52e9d37dd3261d97b6ab7da"}
Nov 03 13:30:03.069748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.069676    5420 scope.go:122] "RemoveContainer" containerID="027f23361f5649e44ccbd04150d852acbbfa2dd1b52e9d37dd3261d97b6ab7da"
Nov 03 13:30:03.075731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.075710    5420 generic.go:356] "Generic (PLEG): container finished" podID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerID="167db8f00b5174724062c837c36d0c50b6efd93c09bb5384ef8a8a111b261a56" exitCode=60
Nov 03 13:30:03.080569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.076289    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" event={"ID":"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8","Type":"ContainerDied","Data":"167db8f00b5174724062c837c36d0c50b6efd93c09bb5384ef8a8a111b261a56"}
Nov 03 13:30:03.080569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.076553    5420 scope.go:122] "RemoveContainer" containerID="167db8f00b5174724062c837c36d0c50b6efd93c09bb5384ef8a8a111b261a56"
Nov 03 13:30:03.161480 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.161432    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" podStartSLOduration=3.16141857 podStartE2EDuration="3.16141857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:03.152413335 +0000 UTC m=+1244.230178367" watchObservedRunningTime="2026-11-03 13:30:03.16141857 +0000 UTC m=+1244.239183596"
Nov 03 13:30:03.686391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.686363    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 13:30:03.686494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.686401    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 13:30:03.736759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:03.735787    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:04.086518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.086487    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" event={"ID":"2c21bb75-b140-4359-af93-f4234cdae01d","Type":"ContainerStarted","Data":"90374a6079caca2951041848449955fd4941abaa1b03380d9e942b3e6b1394a3"}
Nov 03 13:30:04.088624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.088594    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" event={"ID":"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00","Type":"ContainerStarted","Data":"a888e77bbff23d3f7374d02fffae2ec25abc7041a789e6ceee4ea6c6ce39e6a5"}
Nov 03 13:30:04.090617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.090595    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" event={"ID":"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd","Type":"ContainerStarted","Data":"35999815dfbd70c40c8f9d7ecfda9f20b4df0ab1fdbec9fec22f4298f6daa3e8"}
Nov 03 13:30:04.092620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.092597    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" event={"ID":"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8","Type":"ContainerStarted","Data":"8d7b7aa19b0def241a5f40d692a7714a51df24601caf7f3ea663a195ccf27681"}
Nov 03 13:30:04.094854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.094834    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" event={"ID":"f5ceef64-17be-4b59-a562-da452c10930b","Type":"ContainerStarted","Data":"2a26f36afdf3cb8b23f49588356639be613657e5ab9b9a3dabba7024230bc62d"}
Nov 03 13:30:04.096498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.096480    5420 generic.go:356] "Generic (PLEG): container finished" podID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerID="98f6b60fad2d27b960771c4d76c73adc38deb6e07687c457abae9db1ec6e14f7" exitCode=60
Nov 03 13:30:04.096566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.096554    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" event={"ID":"c6624d06-66d9-40b9-8e96-7f9018f66bce","Type":"ContainerDied","Data":"98f6b60fad2d27b960771c4d76c73adc38deb6e07687c457abae9db1ec6e14f7"}
Nov 03 13:30:04.096601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.096583    5420 scope.go:122] "RemoveContainer" containerID="b1add12d97d0978019adc608f946af01b726a00e68a54ba53a82a459274554e7"
Nov 03 13:30:04.096843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.096831    5420 scope.go:122] "RemoveContainer" containerID="98f6b60fad2d27b960771c4d76c73adc38deb6e07687c457abae9db1ec6e14f7"
Nov 03 13:30:04.097063 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:04.097047    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8_odh-ai-gateway-infra(c6624d06-66d9-40b9-8e96-7f9018f66bce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce"
Nov 03 13:30:04.099336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.099314    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" event={"ID":"2aaea81a-1f45-4262-94a5-ae0b5bf486c1","Type":"ContainerStarted","Data":"c7132c57859b4aad37f136a888c03df986f7ac8674aafb8ce67a20332d94e11a"}
Nov 03 13:30:04.101714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.101695    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" event={"ID":"20c6bc18-88af-45d7-a76d-d5a1e683cdba","Type":"ContainerStarted","Data":"9c3368284346a9fd74c98f67824a2299b9be01cca253d21e9235c12ed841d2ff"}
Nov 03 13:30:04.106308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.106271    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" event={"ID":"55f2f37b-476c-4ddb-b013-dfc7d187a74c","Type":"ContainerStarted","Data":"387ecb399a0a1cae070d308cb0e8a47ed2c09829b588cd59a37719832573e48d"}
Nov 03 13:30:04.158731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.158686    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" podStartSLOduration=4.158671403 podStartE2EDuration="4.158671403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:04.103713602 +0000 UTC m=+1245.181478629" watchObservedRunningTime="2026-11-03 13:30:04.158671403 +0000 UTC m=+1245.236436454"
Nov 03 13:30:04.159252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.158933    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" podStartSLOduration=4.158925947 podStartE2EDuration="4.158925947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:04.150900193 +0000 UTC m=+1245.228665220" watchObservedRunningTime="2026-11-03 13:30:04.158925947 +0000 UTC m=+1245.236690973"
Nov 03 13:30:04.194051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.194004    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" podStartSLOduration=4.193990717 podStartE2EDuration="4.193990717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:04.193050248 +0000 UTC m=+1245.270815274" watchObservedRunningTime="2026-11-03 13:30:04.193990717 +0000 UTC m=+1245.271755743"
Nov 03 13:30:04.213547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.213508    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" podStartSLOduration=4.213494787 podStartE2EDuration="4.213494787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:04.208960811 +0000 UTC m=+1245.286725837" watchObservedRunningTime="2026-11-03 13:30:04.213494787 +0000 UTC m=+1245.291259812"
Nov 03 13:30:04.258327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.258278    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" podStartSLOduration=4.258264925 podStartE2EDuration="4.258264925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:04.253896493 +0000 UTC m=+1245.331661520" watchObservedRunningTime="2026-11-03 13:30:04.258264925 +0000 UTC m=+1245.336029951"
Nov 03 13:30:04.557018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.556982    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:04.686456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.686427    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 13:30:04.686625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:04.686476    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 13:30:05.116904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.116874    5420 scope.go:122] "RemoveContainer" containerID="98f6b60fad2d27b960771c4d76c73adc38deb6e07687c457abae9db1ec6e14f7"
Nov 03 13:30:05.117435 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:05.117288    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8_odh-ai-gateway-infra(c6624d06-66d9-40b9-8e96-7f9018f66bce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce"
Nov 03 13:30:05.118785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.118759    5420 generic.go:356] "Generic (PLEG): container finished" podID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerID="c7132c57859b4aad37f136a888c03df986f7ac8674aafb8ce67a20332d94e11a" exitCode=60
Nov 03 13:30:05.119073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.119051    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" event={"ID":"2aaea81a-1f45-4262-94a5-ae0b5bf486c1","Type":"ContainerDied","Data":"c7132c57859b4aad37f136a888c03df986f7ac8674aafb8ce67a20332d94e11a"}
Nov 03 13:30:05.119630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.119110    5420 scope.go:122] "RemoveContainer" containerID="aabf8f4dfb17fc6e5e3398c8c794842df3b5113d76e7967146a1cec63fec1024"
Nov 03 13:30:05.119630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.119302    5420 scope.go:122] "RemoveContainer" containerID="c7132c57859b4aad37f136a888c03df986f7ac8674aafb8ce67a20332d94e11a"
Nov 03 13:30:05.119718 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:05.119649    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw_odh-ai-gateway-infra(2aaea81a-1f45-4262-94a5-ae0b5bf486c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1"
Nov 03 13:30:05.145973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.145944    5420 generic.go:356] "Generic (PLEG): container finished" podID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerID="9c3368284346a9fd74c98f67824a2299b9be01cca253d21e9235c12ed841d2ff" exitCode=60
Nov 03 13:30:05.146132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.146082    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" event={"ID":"20c6bc18-88af-45d7-a76d-d5a1e683cdba","Type":"ContainerDied","Data":"9c3368284346a9fd74c98f67824a2299b9be01cca253d21e9235c12ed841d2ff"}
Nov 03 13:30:05.147669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.146962    5420 scope.go:122] "RemoveContainer" containerID="9c3368284346a9fd74c98f67824a2299b9be01cca253d21e9235c12ed841d2ff"
Nov 03 13:30:05.147669 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:05.147314    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7_odh-ai-gateway-infra(20c6bc18-88af-45d7-a76d-d5a1e683cdba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba"
Nov 03 13:30:05.149603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.149571    5420 generic.go:356] "Generic (PLEG): container finished" podID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerID="387ecb399a0a1cae070d308cb0e8a47ed2c09829b588cd59a37719832573e48d" exitCode=60
Nov 03 13:30:05.149699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.149669    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" event={"ID":"55f2f37b-476c-4ddb-b013-dfc7d187a74c","Type":"ContainerDied","Data":"387ecb399a0a1cae070d308cb0e8a47ed2c09829b588cd59a37719832573e48d"}
Nov 03 13:30:05.151164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.151143    5420 scope.go:122] "RemoveContainer" containerID="387ecb399a0a1cae070d308cb0e8a47ed2c09829b588cd59a37719832573e48d"
Nov 03 13:30:05.151529 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:05.151507    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl_odh-ai-gateway-infra(55f2f37b-476c-4ddb-b013-dfc7d187a74c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c"
Nov 03 13:30:05.152454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.152389    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c21bb75-b140-4359-af93-f4234cdae01d" containerID="90374a6079caca2951041848449955fd4941abaa1b03380d9e942b3e6b1394a3" exitCode=60
Nov 03 13:30:05.152550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.152494    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" event={"ID":"2c21bb75-b140-4359-af93-f4234cdae01d","Type":"ContainerDied","Data":"90374a6079caca2951041848449955fd4941abaa1b03380d9e942b3e6b1394a3"}
Nov 03 13:30:05.153111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.153092    5420 scope.go:122] "RemoveContainer" containerID="90374a6079caca2951041848449955fd4941abaa1b03380d9e942b3e6b1394a3"
Nov 03 13:30:05.153417 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:05.153392    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh_odh-ai-gateway-infra(2c21bb75-b140-4359-af93-f4234cdae01d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" podUID="2c21bb75-b140-4359-af93-f4234cdae01d"
Nov 03 13:30:05.155954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.155934    5420 generic.go:356] "Generic (PLEG): container finished" podID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerID="a888e77bbff23d3f7374d02fffae2ec25abc7041a789e6ceee4ea6c6ce39e6a5" exitCode=60
Nov 03 13:30:05.156050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.155999    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" event={"ID":"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00","Type":"ContainerDied","Data":"a888e77bbff23d3f7374d02fffae2ec25abc7041a789e6ceee4ea6c6ce39e6a5"}
Nov 03 13:30:05.156285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.156267    5420 scope.go:122] "RemoveContainer" containerID="a888e77bbff23d3f7374d02fffae2ec25abc7041a789e6ceee4ea6c6ce39e6a5"
Nov 03 13:30:05.156567 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:05.156548    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg_odh-ai-gateway-infra(2fe356c0-c8f6-4b0e-b9fe-9a1946742d00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00"
Nov 03 13:30:05.159332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.158709    5420 generic.go:356] "Generic (PLEG): container finished" podID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerID="35999815dfbd70c40c8f9d7ecfda9f20b4df0ab1fdbec9fec22f4298f6daa3e8" exitCode=60
Nov 03 13:30:05.159332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.158779    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" event={"ID":"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd","Type":"ContainerDied","Data":"35999815dfbd70c40c8f9d7ecfda9f20b4df0ab1fdbec9fec22f4298f6daa3e8"}
Nov 03 13:30:05.159332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.158973    5420 scope.go:122] "RemoveContainer" containerID="35999815dfbd70c40c8f9d7ecfda9f20b4df0ab1fdbec9fec22f4298f6daa3e8"
Nov 03 13:30:05.159332 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:05.159199    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk_odh-ai-gateway-infra(6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd"
Nov 03 13:30:05.160802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.160777    5420 generic.go:356] "Generic (PLEG): container finished" podID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerID="8d7b7aa19b0def241a5f40d692a7714a51df24601caf7f3ea663a195ccf27681" exitCode=60
Nov 03 13:30:05.160919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.160844    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" event={"ID":"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8","Type":"ContainerDied","Data":"8d7b7aa19b0def241a5f40d692a7714a51df24601caf7f3ea663a195ccf27681"}
Nov 03 13:30:05.161068 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.161055    5420 scope.go:122] "RemoveContainer" containerID="8d7b7aa19b0def241a5f40d692a7714a51df24601caf7f3ea663a195ccf27681"
Nov 03 13:30:05.161343 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:05.161319    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627_odh-ai-gateway-infra(6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8"
Nov 03 13:30:05.162323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.162311    5420 scope.go:122] "RemoveContainer" containerID="74777ea2522b002cf206101217bf09448412dd9abfca4f2c5a4201839983db95"
Nov 03 13:30:05.162916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.162895    5420 generic.go:356] "Generic (PLEG): container finished" podID="f5ceef64-17be-4b59-a562-da452c10930b" containerID="2a26f36afdf3cb8b23f49588356639be613657e5ab9b9a3dabba7024230bc62d" exitCode=60
Nov 03 13:30:05.163008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.162927    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" event={"ID":"f5ceef64-17be-4b59-a562-da452c10930b","Type":"ContainerDied","Data":"2a26f36afdf3cb8b23f49588356639be613657e5ab9b9a3dabba7024230bc62d"}
Nov 03 13:30:05.163199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.163176    5420 scope.go:122] "RemoveContainer" containerID="2a26f36afdf3cb8b23f49588356639be613657e5ab9b9a3dabba7024230bc62d"
Nov 03 13:30:05.163686 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:05.163667    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct_odh-ai-gateway-infra(f5ceef64-17be-4b59-a562-da452c10930b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" podUID="f5ceef64-17be-4b59-a562-da452c10930b"
Nov 03 13:30:05.216426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.216392    5420 scope.go:122] "RemoveContainer" containerID="b14e2efe5b0dcb3356d6269afd29acff8d261f7b39971982a42f820ccbfcd9eb"
Nov 03 13:30:05.258979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.258951    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:05.272927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.272907    5420 scope.go:122] "RemoveContainer" containerID="3ab6377aefb3b5576d7f1c98cc97ef5061995bf05f77f1d44f4ac91b5478cd6c"
Nov 03 13:30:05.289673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.289648    5420 scope.go:122] "RemoveContainer" containerID="748ccf4f90e4313ef7f6c867adbeae3a8b82b86b9856ba218497438e6808da3a"
Nov 03 13:30:05.308145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.308124    5420 scope.go:122] "RemoveContainer" containerID="027f23361f5649e44ccbd04150d852acbbfa2dd1b52e9d37dd3261d97b6ab7da"
Nov 03 13:30:05.356907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.356885    5420 scope.go:122] "RemoveContainer" containerID="167db8f00b5174724062c837c36d0c50b6efd93c09bb5384ef8a8a111b261a56"
Nov 03 13:30:05.373617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.373594    5420 scope.go:122] "RemoveContainer" containerID="fcdd3f4f0cab7ce8b30c84c24d33385d336fd58147f42d2e1ebaa6bdc5c1d42a"
Nov 03 13:30:05.686142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.686059    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 13:30:05.686142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.686105    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 13:30:05.937949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:05.937859    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:06.174075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.174040    5420 scope.go:122] "RemoveContainer" containerID="387ecb399a0a1cae070d308cb0e8a47ed2c09829b588cd59a37719832573e48d"
Nov 03 13:30:06.174530 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:06.174414    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl_odh-ai-gateway-infra(55f2f37b-476c-4ddb-b013-dfc7d187a74c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c"
Nov 03 13:30:06.175821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.175796    5420 scope.go:122] "RemoveContainer" containerID="90374a6079caca2951041848449955fd4941abaa1b03380d9e942b3e6b1394a3"
Nov 03 13:30:06.176131 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:06.176111    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh_odh-ai-gateway-infra(2c21bb75-b140-4359-af93-f4234cdae01d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" podUID="2c21bb75-b140-4359-af93-f4234cdae01d"
Nov 03 13:30:06.177417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.177396    5420 scope.go:122] "RemoveContainer" containerID="a888e77bbff23d3f7374d02fffae2ec25abc7041a789e6ceee4ea6c6ce39e6a5"
Nov 03 13:30:06.177634 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:06.177617    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg_odh-ai-gateway-infra(2fe356c0-c8f6-4b0e-b9fe-9a1946742d00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00"
Nov 03 13:30:06.178977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.178958    5420 scope.go:122] "RemoveContainer" containerID="35999815dfbd70c40c8f9d7ecfda9f20b4df0ab1fdbec9fec22f4298f6daa3e8"
Nov 03 13:30:06.179264 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:06.179244    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk_odh-ai-gateway-infra(6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd"
Nov 03 13:30:06.180583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.180567    5420 scope.go:122] "RemoveContainer" containerID="8d7b7aa19b0def241a5f40d692a7714a51df24601caf7f3ea663a195ccf27681"
Nov 03 13:30:06.180795 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:06.180778    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627_odh-ai-gateway-infra(6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8"
Nov 03 13:30:06.182147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.182128    5420 scope.go:122] "RemoveContainer" containerID="2a26f36afdf3cb8b23f49588356639be613657e5ab9b9a3dabba7024230bc62d"
Nov 03 13:30:06.182418 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:06.182387    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct_odh-ai-gateway-infra(f5ceef64-17be-4b59-a562-da452c10930b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" podUID="f5ceef64-17be-4b59-a562-da452c10930b"
Nov 03 13:30:06.183688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.183668    5420 scope.go:122] "RemoveContainer" containerID="c7132c57859b4aad37f136a888c03df986f7ac8674aafb8ce67a20332d94e11a"
Nov 03 13:30:06.183911 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:06.183885    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw_odh-ai-gateway-infra(2aaea81a-1f45-4262-94a5-ae0b5bf486c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1"
Nov 03 13:30:06.185429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.185410    5420 scope.go:122] "RemoveContainer" containerID="9c3368284346a9fd74c98f67824a2299b9be01cca253d21e9235c12ed841d2ff"
Nov 03 13:30:06.185640 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:06.185620    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7_odh-ai-gateway-infra(20c6bc18-88af-45d7-a76d-d5a1e683cdba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba"
Nov 03 13:30:06.667237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.666840    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:06.692900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.687704    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 13:30:06.692900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:06.687762    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 13:30:07.263235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:07.261876    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:07.686378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:07.686282    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 13:30:07.686378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:07.686340    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 13:30:08.035598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:08.035561    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:08.691141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:08.691110    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:08.691681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:08.691666    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:08.692491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:08.692467    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:30:08.827148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:08.827118    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:09.619235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:09.613177    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:10.208267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:10.208202    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:10.858448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:10.858414    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:11.548574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:11.548543    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:12.207686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:12.206111    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:12.825948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:12.825914    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:13.529373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:13.529304    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:13.865623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:13.865517    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:14.033891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:14.033853    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:14.077744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:14.077719    5420 scope.go:122] "RemoveContainer" containerID="98f6b60fad2d27b960771c4d76c73adc38deb6e07687c457abae9db1ec6e14f7"
Nov 03 13:30:14.918377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:14.918293    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:15.281788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:15.281756    5420 generic.go:356] "Generic (PLEG): container finished" podID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerID="55241f282e114b3c6fd5611948c57fac5e2267ac2db53a694eda39b908abc81b" exitCode=60
Nov 03 13:30:15.281980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:15.281828    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" event={"ID":"c6624d06-66d9-40b9-8e96-7f9018f66bce","Type":"ContainerDied","Data":"55241f282e114b3c6fd5611948c57fac5e2267ac2db53a694eda39b908abc81b"}
Nov 03 13:30:15.281980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:15.281885    5420 scope.go:122] "RemoveContainer" containerID="98f6b60fad2d27b960771c4d76c73adc38deb6e07687c457abae9db1ec6e14f7"
Nov 03 13:30:15.282351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:15.282332    5420 scope.go:122] "RemoveContainer" containerID="55241f282e114b3c6fd5611948c57fac5e2267ac2db53a694eda39b908abc81b"
Nov 03 13:30:15.282754 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:15.282729    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8_odh-ai-gateway-infra(c6624d06-66d9-40b9-8e96-7f9018f66bce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce"
Nov 03 13:30:15.792842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:15.792806    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:16.077130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.077055    5420 scope.go:122] "RemoveContainer" containerID="35999815dfbd70c40c8f9d7ecfda9f20b4df0ab1fdbec9fec22f4298f6daa3e8"
Nov 03 13:30:16.077521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.077426    5420 scope.go:122] "RemoveContainer" containerID="9c3368284346a9fd74c98f67824a2299b9be01cca253d21e9235c12ed841d2ff"
Nov 03 13:30:16.077595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.077578    5420 scope.go:122] "RemoveContainer" containerID="a888e77bbff23d3f7374d02fffae2ec25abc7041a789e6ceee4ea6c6ce39e6a5"
Nov 03 13:30:16.078680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.078657    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:16.366280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.358565    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"]
Nov 03 13:30:16.366280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.364054    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:16.666253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.666229    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:16.823756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.822594    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6624d06-66d9-40b9-8e96-7f9018f66bce-kube-api-access-rkvx6\" (UniqueName: \"kubernetes.io/projected/c6624d06-66d9-40b9-8e96-7f9018f66bce-kube-api-access-rkvx6\") pod \"c6624d06-66d9-40b9-8e96-7f9018f66bce\" (UID: \"c6624d06-66d9-40b9-8e96-7f9018f66bce\") "
Nov 03 13:30:16.823756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.822649    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6624d06-66d9-40b9-8e96-7f9018f66bce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6624d06-66d9-40b9-8e96-7f9018f66bce-maas-api-service-ca\") pod \"c6624d06-66d9-40b9-8e96-7f9018f66bce\" (UID: \"c6624d06-66d9-40b9-8e96-7f9018f66bce\") "
Nov 03 13:30:16.824509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.824461    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6624d06-66d9-40b9-8e96-7f9018f66bce-maas-api-service-ca" pod "c6624d06-66d9-40b9-8e96-7f9018f66bce" (UID: "c6624d06-66d9-40b9-8e96-7f9018f66bce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:16.825490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.825340    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6624d06-66d9-40b9-8e96-7f9018f66bce-kube-api-access-rkvx6" pod "c6624d06-66d9-40b9-8e96-7f9018f66bce" (UID: "c6624d06-66d9-40b9-8e96-7f9018f66bce"). InnerVolumeSpecName "kube-api-access-rkvx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:16.925095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.925009    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkvx6\" (UniqueName: \"kubernetes.io/projected/c6624d06-66d9-40b9-8e96-7f9018f66bce-kube-api-access-rkvx6\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:16.925095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:16.925041    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6624d06-66d9-40b9-8e96-7f9018f66bce-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:17.088142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.088108    5420 scope.go:122] "RemoveContainer" containerID="90374a6079caca2951041848449955fd4941abaa1b03380d9e942b3e6b1394a3"
Nov 03 13:30:17.271273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.271189    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:17.370795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.370537    5420 generic.go:356] "Generic (PLEG): container finished" podID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerID="cdab81030576e33b695d35a5f26d0efeb2d7363528b4dd792f11d19852702214" exitCode=60
Nov 03 13:30:17.370795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.370740    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" event={"ID":"20c6bc18-88af-45d7-a76d-d5a1e683cdba","Type":"ContainerDied","Data":"cdab81030576e33b695d35a5f26d0efeb2d7363528b4dd792f11d19852702214"}
Nov 03 13:30:17.371315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.371259    5420 scope.go:122] "RemoveContainer" containerID="9c3368284346a9fd74c98f67824a2299b9be01cca253d21e9235c12ed841d2ff"
Nov 03 13:30:17.386265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.378352    5420 scope.go:122] "RemoveContainer" containerID="cdab81030576e33b695d35a5f26d0efeb2d7363528b4dd792f11d19852702214"
Nov 03 13:30:17.386265 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:17.378797    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7_odh-ai-gateway-infra(20c6bc18-88af-45d7-a76d-d5a1e683cdba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba"
Nov 03 13:30:17.397977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.395516    5420 generic.go:356] "Generic (PLEG): container finished" podID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerID="8da0dd4d090537127d5fb73675c02fd7d7cf6f2b706d9e714c16b6be20fc367d" exitCode=60
Nov 03 13:30:17.397977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.395609    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" event={"ID":"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00","Type":"ContainerDied","Data":"8da0dd4d090537127d5fb73675c02fd7d7cf6f2b706d9e714c16b6be20fc367d"}
Nov 03 13:30:17.419240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.417414    5420 scope.go:122] "RemoveContainer" containerID="8da0dd4d090537127d5fb73675c02fd7d7cf6f2b706d9e714c16b6be20fc367d"
Nov 03 13:30:17.419240 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:17.417956    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg_odh-ai-gateway-infra(2fe356c0-c8f6-4b0e-b9fe-9a1946742d00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00"
Nov 03 13:30:17.463370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.446371    5420 generic.go:356] "Generic (PLEG): container finished" podID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerID="8fbc5f4cb11391eb57ea339558b84997d4bd4743c582e4d8fd2d2b84030ca852" exitCode=60
Nov 03 13:30:17.463370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.446618    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" event={"ID":"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd","Type":"ContainerDied","Data":"8fbc5f4cb11391eb57ea339558b84997d4bd4743c582e4d8fd2d2b84030ca852"}
Nov 03 13:30:17.463370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.447201    5420 scope.go:122] "RemoveContainer" containerID="8fbc5f4cb11391eb57ea339558b84997d4bd4743c582e4d8fd2d2b84030ca852"
Nov 03 13:30:17.463370 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:17.447700    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk_odh-ai-gateway-infra(6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd"
Nov 03 13:30:17.463370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.458446    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8" event={"ID":"c6624d06-66d9-40b9-8e96-7f9018f66bce","Type":"ContainerDied","Data":"7688d9d3c84bd54f1a380bb05d52793d794255f2d2bc062324a7ea4624c550b9"}
Nov 03 13:30:17.463370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.458543    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"
Nov 03 13:30:17.476297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.474764    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:17.559124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.557742    5420 scope.go:122] "RemoveContainer" containerID="a888e77bbff23d3f7374d02fffae2ec25abc7041a789e6ceee4ea6c6ce39e6a5"
Nov 03 13:30:17.559451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.558655    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"]
Nov 03 13:30:17.567140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.566696    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a62d22-29895210-dtvk8"]
Nov 03 13:30:17.580086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.580058    5420 scope.go:122] "RemoveContainer" containerID="35999815dfbd70c40c8f9d7ecfda9f20b4df0ab1fdbec9fec22f4298f6daa3e8"
Nov 03 13:30:17.599314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:17.599290    5420 scope.go:122] "RemoveContainer" containerID="55241f282e114b3c6fd5611948c57fac5e2267ac2db53a694eda39b908abc81b"
Nov 03 13:30:18.077083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.077054    5420 scope.go:122] "RemoveContainer" containerID="8d7b7aa19b0def241a5f40d692a7714a51df24601caf7f3ea663a195ccf27681"
Nov 03 13:30:18.077307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.077262    5420 scope.go:122] "RemoveContainer" containerID="387ecb399a0a1cae070d308cb0e8a47ed2c09829b588cd59a37719832573e48d"
Nov 03 13:30:18.077464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.077433    5420 scope.go:122] "RemoveContainer" containerID="c7132c57859b4aad37f136a888c03df986f7ac8674aafb8ce67a20332d94e11a"
Nov 03 13:30:18.077620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.077600    5420 scope.go:122] "RemoveContainer" containerID="2a26f36afdf3cb8b23f49588356639be613657e5ab9b9a3dabba7024230bc62d"
Nov 03 13:30:18.080792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.080774    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:18.231581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.231548    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:18.370476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.370447    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:18.410609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.410578    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"]
Nov 03 13:30:18.464267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.464235    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"]
Nov 03 13:30:18.479332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.479296    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"]
Nov 03 13:30:18.482755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.482729    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c21bb75-b140-4359-af93-f4234cdae01d" containerID="73478f827d7e2dd2f5ee5f49863dc960e38d60feb7cf2ca5b781112634ba694a" exitCode=60
Nov 03 13:30:18.482878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.482782    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" event={"ID":"2c21bb75-b140-4359-af93-f4234cdae01d","Type":"ContainerDied","Data":"73478f827d7e2dd2f5ee5f49863dc960e38d60feb7cf2ca5b781112634ba694a"}
Nov 03 13:30:18.482878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.482819    5420 scope.go:122] "RemoveContainer" containerID="90374a6079caca2951041848449955fd4941abaa1b03380d9e942b3e6b1394a3"
Nov 03 13:30:18.483317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.483276    5420 scope.go:122] "RemoveContainer" containerID="73478f827d7e2dd2f5ee5f49863dc960e38d60feb7cf2ca5b781112634ba694a"
Nov 03 13:30:18.483711 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:18.483675    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh_odh-ai-gateway-infra(2c21bb75-b140-4359-af93-f4234cdae01d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" podUID="2c21bb75-b140-4359-af93-f4234cdae01d"
Nov 03 13:30:18.868364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.868337    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:18.928433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.928402    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:18.934596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.934564    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:18.976362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.976326    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-maas-api-service-ca\") pod \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\" (UID: \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\") "
Nov 03 13:30:18.976534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.976507    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c6bc18-88af-45d7-a76d-d5a1e683cdba-kube-api-access-h6wg2\" (UniqueName: \"kubernetes.io/projected/20c6bc18-88af-45d7-a76d-d5a1e683cdba-kube-api-access-h6wg2\") pod \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\" (UID: \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\") "
Nov 03 13:30:18.976632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.976539    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-kube-api-access-2857k\" (UniqueName: \"kubernetes.io/projected/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-kube-api-access-2857k\") pod \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\" (UID: \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\") "
Nov 03 13:30:18.978025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.977885    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-maas-api-service-ca" pod "6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" (UID: "6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:18.979113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.979061    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-maas-api-service-ca\") pod \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\" (UID: \"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00\") "
Nov 03 13:30:18.979253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.979151    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-kube-api-access-c4zbj\") pod \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\" (UID: \"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd\") "
Nov 03 13:30:18.979253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.979189    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c6bc18-88af-45d7-a76d-d5a1e683cdba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c6bc18-88af-45d7-a76d-d5a1e683cdba-maas-api-service-ca\") pod \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\" (UID: \"20c6bc18-88af-45d7-a76d-d5a1e683cdba\") "
Nov 03 13:30:18.979531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.979501    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-maas-api-service-ca" pod "2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" (UID: "2fe356c0-c8f6-4b0e-b9fe-9a1946742d00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:18.979742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.979715    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c6bc18-88af-45d7-a76d-d5a1e683cdba-kube-api-access-h6wg2" pod "20c6bc18-88af-45d7-a76d-d5a1e683cdba" (UID: "20c6bc18-88af-45d7-a76d-d5a1e683cdba"). InnerVolumeSpecName "kube-api-access-h6wg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:18.979930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.979906    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c6bc18-88af-45d7-a76d-d5a1e683cdba-maas-api-service-ca" pod "20c6bc18-88af-45d7-a76d-d5a1e683cdba" (UID: "20c6bc18-88af-45d7-a76d-d5a1e683cdba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:18.980242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.980139    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:18.980242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.980171    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c6bc18-88af-45d7-a76d-d5a1e683cdba-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:18.980242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.980186    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:18.980242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.980199    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6wg2\" (UniqueName: \"kubernetes.io/projected/20c6bc18-88af-45d7-a76d-d5a1e683cdba-kube-api-access-h6wg2\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:18.981415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.981392    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-kube-api-access-2857k" pod "2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" (UID: "2fe356c0-c8f6-4b0e-b9fe-9a1946742d00"). InnerVolumeSpecName "kube-api-access-2857k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:18.981769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:18.981743    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-kube-api-access-c4zbj" pod "6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" (UID: "6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd"). InnerVolumeSpecName "kube-api-access-c4zbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:19.082255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.082205    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd-kube-api-access-c4zbj\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:19.082255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.082256    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2857k\" (UniqueName: \"kubernetes.io/projected/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00-kube-api-access-2857k\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:19.086553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.086524    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" path="/var/lib/kubelet/pods/c6624d06-66d9-40b9-8e96-7f9018f66bce/volumes"
Nov 03 13:30:19.164459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.154740    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:19.495022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.494994    5420 generic.go:356] "Generic (PLEG): container finished" podID="f5ceef64-17be-4b59-a562-da452c10930b" containerID="4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33" exitCode=60
Nov 03 13:30:19.495504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.495067    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" event={"ID":"f5ceef64-17be-4b59-a562-da452c10930b","Type":"ContainerDied","Data":"4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33"}
Nov 03 13:30:19.495504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.495122    5420 scope.go:122] "RemoveContainer" containerID="2a26f36afdf3cb8b23f49588356639be613657e5ab9b9a3dabba7024230bc62d"
Nov 03 13:30:19.495628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.495533    5420 scope.go:122] "RemoveContainer" containerID="4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33"
Nov 03 13:30:19.495930 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:19.495904    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct_odh-ai-gateway-infra(f5ceef64-17be-4b59-a562-da452c10930b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" podUID="f5ceef64-17be-4b59-a562-da452c10930b"
Nov 03 13:30:19.497570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.497549    5420 generic.go:356] "Generic (PLEG): container finished" podID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerID="f7b3c7e634d674832f36af85ea0f06cb95e62f6563ea85b4254bf4dbd02d2952" exitCode=60
Nov 03 13:30:19.497666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.497619    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" event={"ID":"2aaea81a-1f45-4262-94a5-ae0b5bf486c1","Type":"ContainerDied","Data":"f7b3c7e634d674832f36af85ea0f06cb95e62f6563ea85b4254bf4dbd02d2952"}
Nov 03 13:30:19.497984 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.497960    5420 scope.go:122] "RemoveContainer" containerID="f7b3c7e634d674832f36af85ea0f06cb95e62f6563ea85b4254bf4dbd02d2952"
Nov 03 13:30:19.498386 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:19.498369    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw_odh-ai-gateway-infra(2aaea81a-1f45-4262-94a5-ae0b5bf486c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1"
Nov 03 13:30:19.499127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.499114    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"
Nov 03 13:30:19.499203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.499143    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7" event={"ID":"20c6bc18-88af-45d7-a76d-d5a1e683cdba","Type":"ContainerDied","Data":"799ea4e998b5cd3b27697732f3d7005018f758162c92b1afc8420d5cc80b97f5"}
Nov 03 13:30:19.501201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.501181    5420 generic.go:356] "Generic (PLEG): container finished" podID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerID="a1cf9896335058282ff335e90a67a16ed23b338dbf74e715cd82da34debf5826" exitCode=60
Nov 03 13:30:19.501331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.501269    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" event={"ID":"55f2f37b-476c-4ddb-b013-dfc7d187a74c","Type":"ContainerDied","Data":"a1cf9896335058282ff335e90a67a16ed23b338dbf74e715cd82da34debf5826"}
Nov 03 13:30:19.501802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.501704    5420 scope.go:122] "RemoveContainer" containerID="a1cf9896335058282ff335e90a67a16ed23b338dbf74e715cd82da34debf5826"
Nov 03 13:30:19.502054 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:19.502036    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl_odh-ai-gateway-infra(55f2f37b-476c-4ddb-b013-dfc7d187a74c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c"
Nov 03 13:30:19.504681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.504660    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"
Nov 03 13:30:19.504681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.504666    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg" event={"ID":"2fe356c0-c8f6-4b0e-b9fe-9a1946742d00","Type":"ContainerDied","Data":"9312f87e5d930fc17d7ce79ce045a21b849e4a83bd7629de1456b9f817eeeeeb"}
Nov 03 13:30:19.506131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.506110    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk" event={"ID":"6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd","Type":"ContainerDied","Data":"3fe9a134f3db98118ec00778f94b3ff1e735414993d7e0daeaa58bb8c806acc4"}
Nov 03 13:30:19.506228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.506114    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"
Nov 03 13:30:19.508127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.508109    5420 generic.go:356] "Generic (PLEG): container finished" podID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerID="908fd318bd146e3a2aaa7333befb2966be2a341159f6ecbb4e238e293cead918" exitCode=60
Nov 03 13:30:19.508227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.508178    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" event={"ID":"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8","Type":"ContainerDied","Data":"908fd318bd146e3a2aaa7333befb2966be2a341159f6ecbb4e238e293cead918"}
Nov 03 13:30:19.508504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.508485    5420 scope.go:122] "RemoveContainer" containerID="908fd318bd146e3a2aaa7333befb2966be2a341159f6ecbb4e238e293cead918"
Nov 03 13:30:19.508770 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:19.508748    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627_odh-ai-gateway-infra(6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8"
Nov 03 13:30:19.512572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.512557    5420 scope.go:122] "RemoveContainer" containerID="c7132c57859b4aad37f136a888c03df986f7ac8674aafb8ce67a20332d94e11a"
Nov 03 13:30:19.525283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.525263    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"]
Nov 03 13:30:19.528267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.528248    5420 scope.go:122] "RemoveContainer" containerID="cdab81030576e33b695d35a5f26d0efeb2d7363528b4dd792f11d19852702214"
Nov 03 13:30:19.539598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.539567    5420 scope.go:122] "RemoveContainer" containerID="387ecb399a0a1cae070d308cb0e8a47ed2c09829b588cd59a37719832573e48d"
Nov 03 13:30:19.555607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.555584    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"]
Nov 03 13:30:19.562794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.562770    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef1693-29895210-p5xk7"]
Nov 03 13:30:19.563782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.563766    5420 scope.go:122] "RemoveContainer" containerID="8da0dd4d090537127d5fb73675c02fd7d7cf6f2b706d9e714c16b6be20fc367d"
Nov 03 13:30:19.576292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.576266    5420 scope.go:122] "RemoveContainer" containerID="8fbc5f4cb11391eb57ea339558b84997d4bd4743c582e4d8fd2d2b84030ca852"
Nov 03 13:30:19.586886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.586870    5420 scope.go:122] "RemoveContainer" containerID="8d7b7aa19b0def241a5f40d692a7714a51df24601caf7f3ea663a195ccf27681"
Nov 03 13:30:19.621720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.619040    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"]
Nov 03 13:30:19.629207 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.629181    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d3ca48-29895210-8ddsk"]
Nov 03 13:30:19.637919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.637896    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"]
Nov 03 13:30:19.643024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.642999    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895210-z49dg"]
Nov 03 13:30:19.718427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.718404    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:19.798317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.798207    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c21bb75-b140-4359-af93-f4234cdae01d-kube-api-access-cdh24\" (UniqueName: \"kubernetes.io/projected/2c21bb75-b140-4359-af93-f4234cdae01d-kube-api-access-cdh24\") pod \"2c21bb75-b140-4359-af93-f4234cdae01d\" (UID: \"2c21bb75-b140-4359-af93-f4234cdae01d\") "
Nov 03 13:30:19.798454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.798411    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c21bb75-b140-4359-af93-f4234cdae01d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21bb75-b140-4359-af93-f4234cdae01d-maas-api-service-ca\") pod \"2c21bb75-b140-4359-af93-f4234cdae01d\" (UID: \"2c21bb75-b140-4359-af93-f4234cdae01d\") "
Nov 03 13:30:19.798759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.798729    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c21bb75-b140-4359-af93-f4234cdae01d-maas-api-service-ca" pod "2c21bb75-b140-4359-af93-f4234cdae01d" (UID: "2c21bb75-b140-4359-af93-f4234cdae01d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:19.798948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.798931    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c21bb75-b140-4359-af93-f4234cdae01d-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:19.800320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.800301    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c21bb75-b140-4359-af93-f4234cdae01d-kube-api-access-cdh24" pod "2c21bb75-b140-4359-af93-f4234cdae01d" (UID: "2c21bb75-b140-4359-af93-f4234cdae01d"). InnerVolumeSpecName "kube-api-access-cdh24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:19.894001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.893973    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:19.900202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:19.900175    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdh24\" (UniqueName: \"kubernetes.io/projected/2c21bb75-b140-4359-af93-f4234cdae01d-kube-api-access-cdh24\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:20.542755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.541852    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:20.550890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.550864    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"]
Nov 03 13:30:20.554957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.554878    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"
Nov 03 13:30:20.554957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.554898    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh" event={"ID":"2c21bb75-b140-4359-af93-f4234cdae01d","Type":"ContainerDied","Data":"b5f0c9dd10c1ec0b7574af03751898ffc52096faa89f5a2c939426bcebdd14e1"}
Nov 03 13:30:20.555183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.554968    5420 scope.go:122] "RemoveContainer" containerID="73478f827d7e2dd2f5ee5f49863dc960e38d60feb7cf2ca5b781112634ba694a"
Nov 03 13:30:20.566340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.566009    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"]
Nov 03 13:30:20.592838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.592716    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"]
Nov 03 13:30:20.625918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.625886    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"]
Nov 03 13:30:20.655428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.654722    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"]
Nov 03 13:30:20.668175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.668148    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c381a7-29895210-hkmvh"]
Nov 03 13:30:20.823315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.822892    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:20.907475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.907453    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:20.910818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.910799    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:20.914996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.914978    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:20.966333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.966302    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ceef64-17be-4b59-a562-da452c10930b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceef64-17be-4b59-a562-da452c10930b-maas-api-service-ca\") pod \"f5ceef64-17be-4b59-a562-da452c10930b\" (UID: \"f5ceef64-17be-4b59-a562-da452c10930b\") "
Nov 03 13:30:20.966507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.966376    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ceef64-17be-4b59-a562-da452c10930b-kube-api-access-qzbk9\" (UniqueName: \"kubernetes.io/projected/f5ceef64-17be-4b59-a562-da452c10930b-kube-api-access-qzbk9\") pod \"f5ceef64-17be-4b59-a562-da452c10930b\" (UID: \"f5ceef64-17be-4b59-a562-da452c10930b\") "
Nov 03 13:30:20.967313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.967272    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ceef64-17be-4b59-a562-da452c10930b-maas-api-service-ca" pod "f5ceef64-17be-4b59-a562-da452c10930b" (UID: "f5ceef64-17be-4b59-a562-da452c10930b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:20.975332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:20.975299    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ceef64-17be-4b59-a562-da452c10930b-kube-api-access-qzbk9" pod "f5ceef64-17be-4b59-a562-da452c10930b" (UID: "f5ceef64-17be-4b59-a562-da452c10930b"). InnerVolumeSpecName "kube-api-access-qzbk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:21.050103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.050073    5420 scope.go:122] "RemoveContainer" containerID="dcd14727bd19b475cbc7662ce0be2c108261e2d9b2d2ec2fa87d114cc49afdd8"
Nov 03 13:30:21.060665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.060640    5420 scope.go:122] "RemoveContainer" containerID="2e7861a29b44a98c5a89e26f1c188fc21540fe23a534a7e268d09cca01eba67a"
Nov 03 13:30:21.068073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.068045    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f2f37b-476c-4ddb-b013-dfc7d187a74c-kube-api-access-p7mtc\" (UniqueName: \"kubernetes.io/projected/55f2f37b-476c-4ddb-b013-dfc7d187a74c-kube-api-access-p7mtc\") pod \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\" (UID: \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\") "
Nov 03 13:30:21.068199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.068132    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-kube-api-access-cnfsb\" (UniqueName: \"kubernetes.io/projected/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-kube-api-access-cnfsb\") pod \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\" (UID: \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\") "
Nov 03 13:30:21.068199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.068180    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-maas-api-service-ca\") pod \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\" (UID: \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\") "
Nov 03 13:30:21.068341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.068302    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-maas-api-service-ca\") pod \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\" (UID: \"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8\") "
Nov 03 13:30:21.068341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.068336    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f2f37b-476c-4ddb-b013-dfc7d187a74c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f2f37b-476c-4ddb-b013-dfc7d187a74c-maas-api-service-ca\") pod \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\" (UID: \"55f2f37b-476c-4ddb-b013-dfc7d187a74c\") "
Nov 03 13:30:21.068760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.068722    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-maas-api-service-ca" pod "2aaea81a-1f45-4262-94a5-ae0b5bf486c1" (UID: "2aaea81a-1f45-4262-94a5-ae0b5bf486c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:21.068760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.068734    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-maas-api-service-ca" pod "6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" (UID: "6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:21.068972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.068950    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-kube-api-access-pzncw\" (UniqueName: \"kubernetes.io/projected/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-kube-api-access-pzncw\") pod \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\" (UID: \"2aaea81a-1f45-4262-94a5-ae0b5bf486c1\") "
Nov 03 13:30:21.069086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.068950    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f2f37b-476c-4ddb-b013-dfc7d187a74c-maas-api-service-ca" pod "55f2f37b-476c-4ddb-b013-dfc7d187a74c" (UID: "55f2f37b-476c-4ddb-b013-dfc7d187a74c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:30:21.069725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.069701    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:21.069830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.069729    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:21.069830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.069746    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f2f37b-476c-4ddb-b013-dfc7d187a74c-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:21.069830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.069760    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceef64-17be-4b59-a562-da452c10930b-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:21.069830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.069775    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzbk9\" (UniqueName: \"kubernetes.io/projected/f5ceef64-17be-4b59-a562-da452c10930b-kube-api-access-qzbk9\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:21.070687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.070661    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f2f37b-476c-4ddb-b013-dfc7d187a74c-kube-api-access-p7mtc" pod "55f2f37b-476c-4ddb-b013-dfc7d187a74c" (UID: "55f2f37b-476c-4ddb-b013-dfc7d187a74c"). InnerVolumeSpecName "kube-api-access-p7mtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:21.071170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.071145    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-kube-api-access-pzncw" pod "2aaea81a-1f45-4262-94a5-ae0b5bf486c1" (UID: "2aaea81a-1f45-4262-94a5-ae0b5bf486c1"). InnerVolumeSpecName "kube-api-access-pzncw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:21.071757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.071732    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-kube-api-access-cnfsb" pod "6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" (UID: "6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8"). InnerVolumeSpecName "kube-api-access-cnfsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:30:21.071757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.071741    5420 scope.go:122] "RemoveContainer" containerID="c95a13d94f4e8678dcfafd29068fd1c84b82da2b1dfb1e89aa770e7d577a5bdb"
Nov 03 13:30:21.081362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.081345    5420 scope.go:122] "RemoveContainer" containerID="098d6a34d2e99d3cd4ca1b68062f9e5e4fe704aea82fc4dc80da04a78e335f2d"
Nov 03 13:30:21.088353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.088325    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" path="/var/lib/kubelet/pods/20c6bc18-88af-45d7-a76d-d5a1e683cdba/volumes"
Nov 03 13:30:21.089043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.089022    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" path="/var/lib/kubelet/pods/2c21bb75-b140-4359-af93-f4234cdae01d/volumes"
Nov 03 13:30:21.089629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.089610    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" path="/var/lib/kubelet/pods/2fe356c0-c8f6-4b0e-b9fe-9a1946742d00/volumes"
Nov 03 13:30:21.090628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.090610    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" path="/var/lib/kubelet/pods/6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd/volumes"
Nov 03 13:30:21.094026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.094009    5420 scope.go:122] "RemoveContainer" containerID="f91eb848b76e4964343a20744b87071871b52c818bd12af5b71089cca24b10be"
Nov 03 13:30:21.104864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.104844    5420 scope.go:122] "RemoveContainer" containerID="7ad95593e6051305f6551d1f7c03055b1a47aa5c396a82d4d7b20049fa1867b1"
Nov 03 13:30:21.115611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.115595    5420 scope.go:122] "RemoveContainer" containerID="8b54b81a3f03bd101fcbe27c2824a07992d5fd7354c1dd0c429d163fcee4d236"
Nov 03 13:30:21.125005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.124990    5420 scope.go:122] "RemoveContainer" containerID="37730fd654700ac89cb948bcd63a4fa4b1deb76f8afcbb940780d724761a8369"
Nov 03 13:30:21.171451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.171417    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzncw\" (UniqueName: \"kubernetes.io/projected/2aaea81a-1f45-4262-94a5-ae0b5bf486c1-kube-api-access-pzncw\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:21.171528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.171450    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7mtc\" (UniqueName: \"kubernetes.io/projected/55f2f37b-476c-4ddb-b013-dfc7d187a74c-kube-api-access-p7mtc\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:21.171528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.171469    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnfsb\" (UniqueName: \"kubernetes.io/projected/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8-kube-api-access-cnfsb\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:30:21.331852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.331757    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:21.581701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.581674    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"
Nov 03 13:30:21.582201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.581675    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl" event={"ID":"55f2f37b-476c-4ddb-b013-dfc7d187a74c","Type":"ContainerDied","Data":"c8db89fd8bbc4a8c0c737ec1f09fa570ab804c7ba522d38b13fb64f372056991"}
Nov 03 13:30:21.582201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.581882    5420 scope.go:122] "RemoveContainer" containerID="a1cf9896335058282ff335e90a67a16ed23b338dbf74e715cd82da34debf5826"
Nov 03 13:30:21.584150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.584102    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627" event={"ID":"6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8","Type":"ContainerDied","Data":"846f67339ef583f844dc01f98d6ef760e5a5d5e8e5b6f9edba15f177246bee4b"}
Nov 03 13:30:21.584324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.584308    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"
Nov 03 13:30:21.588600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.588576    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"
Nov 03 13:30:21.588868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.588578    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct" event={"ID":"f5ceef64-17be-4b59-a562-da452c10930b","Type":"ContainerDied","Data":"5fcafeea87dad8f1bc1a573c4d0accafc242adbf1b1fe6b71148abc70fe20afc"}
Nov 03 13:30:21.591398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.591036    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw" event={"ID":"2aaea81a-1f45-4262-94a5-ae0b5bf486c1","Type":"ContainerDied","Data":"e60aa7d902a0824caadf379a10ba6878797ec63c9ef79dd8ceb199eb198366ff"}
Nov 03 13:30:21.591398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.591124    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"
Nov 03 13:30:21.602566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.602551    5420 scope.go:122] "RemoveContainer" containerID="908fd318bd146e3a2aaa7333befb2966be2a341159f6ecbb4e238e293cead918"
Nov 03 13:30:21.613099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.613081    5420 scope.go:122] "RemoveContainer" containerID="4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33"
Nov 03 13:30:21.621105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.620967    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"]
Nov 03 13:30:21.625608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.625591    5420 scope.go:122] "RemoveContainer" containerID="f7b3c7e634d674832f36af85ea0f06cb95e62f6563ea85b4254bf4dbd02d2952"
Nov 03 13:30:21.626757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.626738    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1a18c8-29895210-5vcfl"]
Nov 03 13:30:21.640804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.640783    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"]
Nov 03 13:30:21.646991 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.646968    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e440bc-29895210-td627"]
Nov 03 13:30:21.666837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.666808    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"]
Nov 03 13:30:21.673758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.673735    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895210-qqtct"]
Nov 03 13:30:21.685902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.685881    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"]
Nov 03 13:30:21.689972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:21.689951    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895210-cj4vw"]
Nov 03 13:30:22.040233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:22.038930    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:22.477850 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:22.477819    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:30:22.718131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:22.718091    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:23.086512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:23.086483    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" path="/var/lib/kubelet/pods/2aaea81a-1f45-4262-94a5-ae0b5bf486c1/volumes"
Nov 03 13:30:23.086947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:23.086933    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" path="/var/lib/kubelet/pods/55f2f37b-476c-4ddb-b013-dfc7d187a74c/volumes"
Nov 03 13:30:23.087358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:23.087343    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" path="/var/lib/kubelet/pods/6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8/volumes"
Nov 03 13:30:23.088054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:23.088039    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ceef64-17be-4b59-a562-da452c10930b" path="/var/lib/kubelet/pods/f5ceef64-17be-4b59-a562-da452c10930b/volumes"
Nov 03 13:30:23.326941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:23.326862    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:23.568442 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:23.568407    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:23.730235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:23.727239    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:24.558001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:24.557968    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:24.733371 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:24.733334    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:30:25.263243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:25.263182    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:26.024501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:26.024463    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:26.042857 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:26.042817    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:30:26.739141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:26.739097    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:27.326238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:27.319985    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:27.891664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:27.891561    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:28.496383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:28.496350    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:29.100668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:29.100617    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:29.709707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:29.709670    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:30.074322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:30.074244    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:30.219997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:30.219959    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:30.650235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:30.649179    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:31.430566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:31.430533    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:31.955578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:31.955548    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:32.091116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:32.091085    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:32.621291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:32.621258    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:32.791377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:32.791336    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:33.490728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:33.490673    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:33.675964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:33.675928    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:33.926323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:33.914763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:34.853554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:34.853522    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:30:34.970502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:34.970454    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:35.728304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:35.728260    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:36.255434 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:36.255393    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:30:36.652399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:36.652313    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:37.004814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:37.004772    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:37.306999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:37.306916    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:37.407919 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:37.407865    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:30:37.621338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:37.621181    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:38.196976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:38.196925    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:38.844302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:38.843761    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:39.835753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:39.835718    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:40.463366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:40.463326    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:41.016860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:41.016820    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:41.622685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:41.622648    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:42.270852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:42.270779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:42.906604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:42.906572    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:43.625896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:43.625865    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:44.488303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:44.488269    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:45.063332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:45.063300    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:45.246114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:45.246083    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:45.910553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:45.910525    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:46.278468 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:46.278433    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:30:46.325500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:46.325473    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:46.521242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:46.517311    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:46.744540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:46.744508    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:47.402076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:47.402046    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:48.007389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:48.007352    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:48.615571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:48.615517    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:48.967932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:48.967901    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:49.196418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:49.196385    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:49.797936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:49.797881    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:50.026830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:50.026795    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"]
Nov 03 13:30:50.033746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:50.033723    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd278693-z2vrq"]
Nov 03 13:30:50.444514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:50.444485    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:51.037012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:51.036982    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:51.086420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:51.086390    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf206e3f-d781-4e03-ba7a-3c1b481c8230" path="/var/lib/kubelet/pods/cf206e3f-d781-4e03-ba7a-3c1b481c8230/volumes"
Nov 03 13:30:51.727814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:51.727782    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:52.398866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:52.398833    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:52.530226 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:52.530188    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:30:53.119684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:53.119657    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:53.264804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:53.264775    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:54.008808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:54.008778    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:54.643521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:54.643493    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:54.734100 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:54.734063    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:30:55.289727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:55.289692    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:55.519272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:55.515965    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:55.883268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:55.883060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:56.296476 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:30:56.296445    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:30:56.441245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:56.441138    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:57.123461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:57.123428    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:57.732884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:57.732846    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:58.327742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:58.327707    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:59.011165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:59.011129    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:59.638282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:30:59.638247    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:00.291412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:00.291377    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:01.025100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:01.025054    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:01.684716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:01.684680    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:02.240428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:02.240393    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:03.122295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:03.119107    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:04.136237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:04.135971    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:05.100020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:05.099985    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:05.251703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:05.251085    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:05.716791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:05.716763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:06.147848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:06.147761    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:06.309332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:06.309191    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:06.593692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:06.593348    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:31:06.593692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:06.593417    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:31:06.606396 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:06.606370    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:31:06.718604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:06.718541    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:31:06.718604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:06.718601    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:31:07.446798 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:07.446761    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:31:07.869658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:07.869620    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:08.112757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:08.112724    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:09.014854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:09.014805    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:09.649435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:09.649393    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:10.385378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:10.385341    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:11.140842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:11.140804    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:11.943629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:11.943591    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:12.659875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:12.659777    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:13.336880 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:13.334357    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:14.027151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.027103    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:14.151343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.151310    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"]
Nov 03 13:31:14.152009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.151994    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerName="cleanup"
Nov 03 13:31:14.152058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152011    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerName="cleanup"
Nov 03 13:31:14.152058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152022    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" containerName="cleanup"
Nov 03 13:31:14.152058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152027    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" containerName="cleanup"
Nov 03 13:31:14.152058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152038    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerName="cleanup"
Nov 03 13:31:14.152058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152042    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerName="cleanup"
Nov 03 13:31:14.152058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152050    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f5ceef64-17be-4b59-a562-da452c10930b" containerName="cleanup"
Nov 03 13:31:14.152058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152054    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ceef64-17be-4b59-a562-da452c10930b" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152062    5420 cpu_manager.go:414] "containerMap: removing container" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152066    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152072    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152077    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152085    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152090    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152102    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152106    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152118    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152123    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152133    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152137    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152146    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152150    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152157    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152161    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152167    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152170    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152179    5420 cpu_manager.go:414] "containerMap: removing container" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152183    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152190    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152195    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152202    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152220    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152240    5420 cpu_manager.go:414] "containerMap: removing container" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152246    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152254    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152259    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152269    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f5ceef64-17be-4b59-a562-da452c10930b" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152273    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ceef64-17be-4b59-a562-da452c10930b" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152282    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152287    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152298    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152303    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152309    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152314    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152319    5420 cpu_manager.go:414] "containerMap: removing container" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerName="cleanup"
Nov 03 13:31:14.152354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152323    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152437    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152446    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ceef64-17be-4b59-a562-da452c10930b" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152452    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152457    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152462    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152470    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152474    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152480    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152485    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152509    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152516    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152522    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ceef64-17be-4b59-a562-da452c10930b" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152529    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152536    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152541    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6624d06-66d9-40b9-8e96-7f9018f66bce" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152547    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152551    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c21bb75-b140-4359-af93-f4234cdae01d" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152558    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152563    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ceef64-17be-4b59-a562-da452c10930b" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152604    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152612    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerName="cleanup"
Nov 03 13:31:14.153811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.152622    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerName="cleanup"
Nov 03 13:31:14.156299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.156282    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.161587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.161426    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:31:14.161870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.161830    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:31:14.161870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.161844    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:31:14.161870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.161854    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:31:14.164558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.164538    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"]
Nov 03 13:31:14.278747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.278677    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj76x\" (UniqueName: \"kubernetes.io/projected/0dd828ff-07e8-49e8-8359-73395f78b6db-kube-api-access-jj76x\") pod \"maas-api-revoke-sub-004aec0b-7hls6\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.278747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.278720    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-004aec0b-7hls6\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.278932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.278853    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-004aec0b-7hls6\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.380501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.380468    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-004aec0b-7hls6\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.380884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.380692    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj76x\" (UniqueName: \"kubernetes.io/projected/0dd828ff-07e8-49e8-8359-73395f78b6db-kube-api-access-jj76x\") pod \"maas-api-revoke-sub-004aec0b-7hls6\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.380884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.380724    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-004aec0b-7hls6\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.381332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.381310    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-004aec0b-7hls6\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.382749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.382727    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-004aec0b-7hls6\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.389190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.389167    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj76x\" (UniqueName: \"kubernetes.io/projected/0dd828ff-07e8-49e8-8359-73395f78b6db-kube-api-access-jj76x\") pod \"maas-api-revoke-sub-004aec0b-7hls6\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.473485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.473459    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:14.760235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.760146    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"]
Nov 03 13:31:14.760457 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:31:14.760431    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd828ff_07e8_49e8_8359_73395f78b6db.slice/crio-1576f223ee377d717fb22b11a33a09649d8bab9a1a379c4ddd36b99217b14700 WatchSource:0}: Error finding container 1576f223ee377d717fb22b11a33a09649d8bab9a1a379c4ddd36b99217b14700: Status 404 returned error can't find the container with id 1576f223ee377d717fb22b11a33a09649d8bab9a1a379c4ddd36b99217b14700
Nov 03 13:31:14.811411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:14.811385    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:15.368659 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:15.368629    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:31:15.368889 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:15.368660    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:31:15.450312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:15.450201    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6" event={"ID":"0dd828ff-07e8-49e8-8359-73395f78b6db","Type":"ContainerStarted","Data":"0d35bdb40978c459b29fece20557afb1bafc1969f08375cb38500b61278cecff"}
Nov 03 13:31:15.450312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:15.450299    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6" event={"ID":"0dd828ff-07e8-49e8-8359-73395f78b6db","Type":"ContainerStarted","Data":"1576f223ee377d717fb22b11a33a09649d8bab9a1a379c4ddd36b99217b14700"}
Nov 03 13:31:15.467753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:15.467683    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6" podStartSLOduration=1.467667358 podStartE2EDuration="1.467667358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:15.466916226 +0000 UTC m=+1316.544681252" watchObservedRunningTime="2026-11-03 13:31:15.467667358 +0000 UTC m=+1316.545432386"
Nov 03 13:31:15.537928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:15.537898    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:16.168979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:16.168943    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:16.460012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:16.459972    5420 generic.go:356] "Generic (PLEG): container finished" podID="0dd828ff-07e8-49e8-8359-73395f78b6db" containerID="0d35bdb40978c459b29fece20557afb1bafc1969f08375cb38500b61278cecff" exitCode=0
Nov 03 13:31:16.460505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:16.460048    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6" event={"ID":"0dd828ff-07e8-49e8-8359-73395f78b6db","Type":"ContainerDied","Data":"0d35bdb40978c459b29fece20557afb1bafc1969f08375cb38500b61278cecff"}
Nov 03 13:31:16.621446 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:16.621404    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceef64_17be_4b59_a562_da452c10930b.slice/crio-conmon-4c43eb802817d449d8ad59072b2145f49ac985d08266945e0e82a1171a38be33.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:31:16.831817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:16.831732    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:17.412524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.412484    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:17.665686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.665620    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:17.739293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.739259    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd828ff-07e8-49e8-8359-73395f78b6db-kube-api-access-jj76x\" (UniqueName: \"kubernetes.io/projected/0dd828ff-07e8-49e8-8359-73395f78b6db-kube-api-access-jj76x\") pod \"0dd828ff-07e8-49e8-8359-73395f78b6db\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") "
Nov 03 13:31:17.739485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.739410    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-openshift-ca\") pod \"0dd828ff-07e8-49e8-8359-73395f78b6db\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") "
Nov 03 13:31:17.739646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.739625    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-xks-ca\") pod \"0dd828ff-07e8-49e8-8359-73395f78b6db\" (UID: \"0dd828ff-07e8-49e8-8359-73395f78b6db\") "
Nov 03 13:31:17.739889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.739841    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-openshift-ca" pod "0dd828ff-07e8-49e8-8359-73395f78b6db" (UID: "0dd828ff-07e8-49e8-8359-73395f78b6db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:17.740398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.740374    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:17.742067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.742042    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-xks-ca" pod "0dd828ff-07e8-49e8-8359-73395f78b6db" (UID: "0dd828ff-07e8-49e8-8359-73395f78b6db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:31:17.742181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.742047    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd828ff-07e8-49e8-8359-73395f78b6db-kube-api-access-jj76x" pod "0dd828ff-07e8-49e8-8359-73395f78b6db" (UID: "0dd828ff-07e8-49e8-8359-73395f78b6db"). InnerVolumeSpecName "kube-api-access-jj76x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:17.841958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.841931    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj76x\" (UniqueName: \"kubernetes.io/projected/0dd828ff-07e8-49e8-8359-73395f78b6db-kube-api-access-jj76x\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:17.841958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:17.841958    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd828ff-07e8-49e8-8359-73395f78b6db-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:18.037859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.037387    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:18.136997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.136969    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"]
Nov 03 13:31:18.137200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.137179    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy" containerID="cri-o://8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450" gracePeriod=30
Nov 03 13:31:18.141845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.141819    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"]
Nov 03 13:31:18.142017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.141997    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy" containerID="cri-o://919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490" gracePeriod=30
Nov 03 13:31:18.213465 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.213418    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:18.351812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.351724    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:18.480655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.480616    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"
Nov 03 13:31:18.480655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.480650    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6" event={"ID":"0dd828ff-07e8-49e8-8359-73395f78b6db","Type":"ContainerDied","Data":"1576f223ee377d717fb22b11a33a09649d8bab9a1a379c4ddd36b99217b14700"}
Nov 03 13:31:18.480865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.480695    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1576f223ee377d717fb22b11a33a09649d8bab9a1a379c4ddd36b99217b14700"
Nov 03 13:31:18.753295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:18.753182    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:19.354749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:19.354713    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:20.116457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:20.116426    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:20.863229 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:20.863184    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:21.237980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:21.237951    5420 scope.go:122] "RemoveContainer" containerID="62fe23324bdb35855813f7636b65e295eed2d686dcae330780e044c0eb1532d5"
Nov 03 13:31:21.249015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:21.248994    5420 scope.go:122] "RemoveContainer" containerID="e1f8bf557524f499345447bf98132500fbf75f8f66a04cc924760e6f5b1173d0"
Nov 03 13:31:21.258481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:21.258463    5420 scope.go:122] "RemoveContainer" containerID="bcaa0d2658c8a9340b477a21990955ae2cc6bc5f7589f4da502cc0288e1f0d00"
Nov 03 13:31:21.269149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:21.269130    5420 scope.go:122] "RemoveContainer" containerID="731c3ab0468d487788b85a457456f15005a00f616bb1eea6885b1c63efdcb8a7"
Nov 03 13:31:21.632936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:21.619805    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:22.361994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:22.361954    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:23.153283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.152361    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:23.443614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.443582    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:31:23.468197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.468119    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:31:23.528853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.528816    5420 generic.go:356] "Generic (PLEG): container finished" podID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerID="919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490" exitCode=0
Nov 03 13:31:23.529026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.528871    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"
Nov 03 13:31:23.529026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.528868    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" event={"ID":"3de14b3f-b00e-4105-b5e9-4b52ef946958","Type":"ContainerDied","Data":"919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490"}
Nov 03 13:31:23.529159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.529098    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp" event={"ID":"3de14b3f-b00e-4105-b5e9-4b52ef946958","Type":"ContainerDied","Data":"2234b7ec7ed6df1cfd969d098b8b44c2bdec19fcb1eaba0b2e9309675cc65be4"}
Nov 03 13:31:23.529159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.529135    5420 scope.go:122] "RemoveContainer" containerID="919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490"
Nov 03 13:31:23.530699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.530673    5420 generic.go:356] "Generic (PLEG): container finished" podID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerID="8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450" exitCode=0
Nov 03 13:31:23.530827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.530764    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" event={"ID":"25c5bca5-53b9-475e-a4d3-c7fa153c4c35","Type":"ContainerDied","Data":"8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450"}
Nov 03 13:31:23.530827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.530805    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96" event={"ID":"25c5bca5-53b9-475e-a4d3-c7fa153c4c35","Type":"ContainerDied","Data":"eac143864edf9ce696afaeedd7eceae9aedabcdfd8f759d1d9b105aed145d547"}
Nov 03 13:31:23.530954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.530869    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"
Nov 03 13:31:23.531730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.531706    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-kube-api-access-hdc84\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-kube-api-access-hdc84\") pod \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") "
Nov 03 13:31:23.531855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.531809    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-envoy\") pod \"3de14b3f-b00e-4105-b5e9-4b52ef946958\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") "
Nov 03 13:31:23.531855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.531845    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-podinfo\") pod \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") "
Nov 03 13:31:23.531985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.531910    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-envoy\") pod \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") "
Nov 03 13:31:23.532052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532012    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-podinfo\") pod \"3de14b3f-b00e-4105-b5e9-4b52ef946958\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") "
Nov 03 13:31:23.532052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532045    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de14b3f-b00e-4105-b5e9-4b52ef946958-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de14b3f-b00e-4105-b5e9-4b52ef946958-istiod-ca-cert\") pod \"3de14b3f-b00e-4105-b5e9-4b52ef946958\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") "
Nov 03 13:31:23.532162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532081    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-kube-api-access-k2wkq\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-kube-api-access-k2wkq\") pod \"3de14b3f-b00e-4105-b5e9-4b52ef946958\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") "
Nov 03 13:31:23.532162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532139    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-data\") pod \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") "
Nov 03 13:31:23.532300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532166    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-token\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-token\") pod \"3de14b3f-b00e-4105-b5e9-4b52ef946958\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") "
Nov 03 13:31:23.532300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532239    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-credential-socket\") pod \"3de14b3f-b00e-4105-b5e9-4b52ef946958\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") "
Nov 03 13:31:23.532300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532272    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-certs\") pod \"3de14b3f-b00e-4105-b5e9-4b52ef946958\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") "
Nov 03 13:31:23.532459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532352    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istiod-ca-cert\") pod \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") "
Nov 03 13:31:23.532459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532414    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-credential-socket\") pod \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") "
Nov 03 13:31:23.532572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532503    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-certs\") pod \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") "
Nov 03 13:31:23.532572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532532    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-data\") pod \"3de14b3f-b00e-4105-b5e9-4b52ef946958\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") "
Nov 03 13:31:23.532677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532576    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-token\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-token\") pod \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") "
Nov 03 13:31:23.532965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532731    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-socket\") pod \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\" (UID: \"25c5bca5-53b9-475e-a4d3-c7fa153c4c35\") "
Nov 03 13:31:23.532965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.532823    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-socket\") pod \"3de14b3f-b00e-4105-b5e9-4b52ef946958\" (UID: \"3de14b3f-b00e-4105-b5e9-4b52ef946958\") "
Nov 03 13:31:23.533115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.533069    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-credential-socket" pod "3de14b3f-b00e-4105-b5e9-4b52ef946958" (UID: "3de14b3f-b00e-4105-b5e9-4b52ef946958"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.533420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.533182    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istiod-ca-cert" pod "25c5bca5-53b9-475e-a4d3-c7fa153c4c35" (UID: "25c5bca5-53b9-475e-a4d3-c7fa153c4c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:23.533420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.533240    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-data" pod "25c5bca5-53b9-475e-a4d3-c7fa153c4c35" (UID: "25c5bca5-53b9-475e-a4d3-c7fa153c4c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.534789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.533466    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-socket" pod "3de14b3f-b00e-4105-b5e9-4b52ef946958" (UID: "3de14b3f-b00e-4105-b5e9-4b52ef946958"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.534789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.533555    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-certs" pod "3de14b3f-b00e-4105-b5e9-4b52ef946958" (UID: "3de14b3f-b00e-4105-b5e9-4b52ef946958"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.534789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.533661    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-credential-socket" pod "25c5bca5-53b9-475e-a4d3-c7fa153c4c35" (UID: "25c5bca5-53b9-475e-a4d3-c7fa153c4c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.534789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.533851    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-socket" pod "25c5bca5-53b9-475e-a4d3-c7fa153c4c35" (UID: "25c5bca5-53b9-475e-a4d3-c7fa153c4c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.534789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.534185    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-certs" pod "25c5bca5-53b9-475e-a4d3-c7fa153c4c35" (UID: "25c5bca5-53b9-475e-a4d3-c7fa153c4c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.534789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.534509    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-data" pod "3de14b3f-b00e-4105-b5e9-4b52ef946958" (UID: "3de14b3f-b00e-4105-b5e9-4b52ef946958"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.535335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535028    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de14b3f-b00e-4105-b5e9-4b52ef946958-istiod-ca-cert" pod "3de14b3f-b00e-4105-b5e9-4b52ef946958" (UID: "3de14b3f-b00e-4105-b5e9-4b52ef946958"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535773    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535845    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535862    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535875    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535890    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535905    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de14b3f-b00e-4105-b5e9-4b52ef946958-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535918    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535932    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535945    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.535987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.535960    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.536644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.536317    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-token" pod "25c5bca5-53b9-475e-a4d3-c7fa153c4c35" (UID: "25c5bca5-53b9-475e-a4d3-c7fa153c4c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:23.538517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.538417    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-kube-api-access-k2wkq" pod "3de14b3f-b00e-4105-b5e9-4b52ef946958" (UID: "3de14b3f-b00e-4105-b5e9-4b52ef946958"). InnerVolumeSpecName "kube-api-access-k2wkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:23.538517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.538485    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-envoy" pod "25c5bca5-53b9-475e-a4d3-c7fa153c4c35" (UID: "25c5bca5-53b9-475e-a4d3-c7fa153c4c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.538517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.538500    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-token" pod "3de14b3f-b00e-4105-b5e9-4b52ef946958" (UID: "3de14b3f-b00e-4105-b5e9-4b52ef946958"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:23.539173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.539140    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-podinfo" pod "3de14b3f-b00e-4105-b5e9-4b52ef946958" (UID: "3de14b3f-b00e-4105-b5e9-4b52ef946958"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:31:23.539785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.539763    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-podinfo" pod "25c5bca5-53b9-475e-a4d3-c7fa153c4c35" (UID: "25c5bca5-53b9-475e-a4d3-c7fa153c4c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:31:23.540413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.540395    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-kube-api-access-hdc84" pod "25c5bca5-53b9-475e-a4d3-c7fa153c4c35" (UID: "25c5bca5-53b9-475e-a4d3-c7fa153c4c35"). InnerVolumeSpecName "kube-api-access-hdc84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:23.540501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.540456    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-envoy" pod "3de14b3f-b00e-4105-b5e9-4b52ef946958" (UID: "3de14b3f-b00e-4105-b5e9-4b52ef946958"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:31:23.548067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.548046    5420 scope.go:122] "RemoveContainer" containerID="919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490"
Nov 03 13:31:23.548454 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:23.548430    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490\": container with ID starting with 919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490 not found: ID does not exist" containerID="919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490"
Nov 03 13:31:23.548539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.548461    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490"} err="failed to get container status \"919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490\": rpc error: code = NotFound desc = could not find container \"919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490\": container with ID starting with 919b73a60182813467e9689995bb7dda802b39102623f9be294df639677f7490 not found: ID does not exist"
Nov 03 13:31:23.548539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.548481    5420 scope.go:122] "RemoveContainer" containerID="8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450"
Nov 03 13:31:23.558641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.558622    5420 scope.go:122] "RemoveContainer" containerID="8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450"
Nov 03 13:31:23.558963 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:23.558933    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450\": container with ID starting with 8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450 not found: ID does not exist" containerID="8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450"
Nov 03 13:31:23.559030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.558973    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450"} err="failed to get container status \"8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450\": rpc error: code = NotFound desc = could not find container \"8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450\": container with ID starting with 8dbb3529f8a492f989cc481987c8c01a65ba8604a1aec59ac513da872bf50450 not found: ID does not exist"
Nov 03 13:31:23.636936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.636898    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.636936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.636928    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.636936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.636939    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2wkq\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-kube-api-access-k2wkq\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.637172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.636949    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.637172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.636957    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.637172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.636965    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdc84\" (UniqueName: \"kubernetes.io/projected/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-kube-api-access-hdc84\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.637172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.636975    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de14b3f-b00e-4105-b5e9-4b52ef946958-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.637172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.636983    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c5bca5-53b9-475e-a4d3-c7fa153c4c35-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:23.858287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.858257    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"]
Nov 03 13:31:23.863548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.863518    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-rfvzp"]
Nov 03 13:31:23.874671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.874643    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"]
Nov 03 13:31:23.882947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:23.882920    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-4gj96"]
Nov 03 13:31:24.739193 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:24.739150    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:31:24.780225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780183    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"]
Nov 03 13:31:24.780893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780878    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f5ceef64-17be-4b59-a562-da452c10930b" containerName="cleanup"
Nov 03 13:31:24.780947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780895    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ceef64-17be-4b59-a562-da452c10930b" containerName="cleanup"
Nov 03 13:31:24.780947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780906    5420 cpu_manager.go:414] "containerMap: removing container" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerName="cleanup"
Nov 03 13:31:24.780947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780912    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c6bc18-88af-45d7-a76d-d5a1e683cdba" containerName="cleanup"
Nov 03 13:31:24.780947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780921    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0dd828ff-07e8-49e8-8359-73395f78b6db" containerName="revoke-keys"
Nov 03 13:31:24.780947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780927    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd828ff-07e8-49e8-8359-73395f78b6db" containerName="revoke-keys"
Nov 03 13:31:24.780947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780935    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerName="cleanup"
Nov 03 13:31:24.780947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780940    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerName="cleanup"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780963    5420 cpu_manager.go:414] "containerMap: removing container" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780968    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780987    5420 cpu_manager.go:414] "containerMap: removing container" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerName="cleanup"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.780992    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerName="cleanup"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781000    5420 cpu_manager.go:414] "containerMap: removing container" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781005    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781111    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aaea81a-1f45-4262-94a5-ae0b5bf486c1" containerName="cleanup"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781119    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd828ff-07e8-49e8-8359-73395f78b6db" containerName="revoke-keys"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781130    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" containerName="istio-proxy"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781136    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6550bebc-8bb9-4d87-88cc-e4ad7c2f17cd" containerName="cleanup"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781144    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9ea2ad-2eba-4bcb-a60b-5d806b4e5bb8" containerName="cleanup"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781151    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe356c0-c8f6-4b0e-b9fe-9a1946742d00" containerName="cleanup"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781157    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f2f37b-476c-4ddb-b013-dfc7d187a74c" containerName="cleanup"
Nov 03 13:31:24.781172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.781164    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" containerName="istio-proxy"
Nov 03 13:31:24.786686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.786659    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:24.790494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.790472    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:31:24.792036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.792015    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:31:24.792160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.792043    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:31:24.792160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.792053    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:31:24.794573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.794552    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"]
Nov 03 13:31:24.863027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.862998    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75212753-hv52l\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:24.863186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.863081    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75212753-hv52l\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:24.863282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.863263    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xsg\" (UniqueName: \"kubernetes.io/projected/b1420976-939e-4a5a-96db-c41d7c234201-kube-api-access-m5xsg\") pod \"maas-api-revoke-sub-75212753-hv52l\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:24.964399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.964365    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xsg\" (UniqueName: \"kubernetes.io/projected/b1420976-939e-4a5a-96db-c41d7c234201-kube-api-access-m5xsg\") pod \"maas-api-revoke-sub-75212753-hv52l\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:24.964570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.964512    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75212753-hv52l\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:24.964633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.964578    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75212753-hv52l\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:24.965252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.965205    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75212753-hv52l\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:24.967406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.967382    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75212753-hv52l\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:24.972904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:24.972874    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xsg\" (UniqueName: \"kubernetes.io/projected/b1420976-939e-4a5a-96db-c41d7c234201-kube-api-access-m5xsg\") pod \"maas-api-revoke-sub-75212753-hv52l\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:25.085764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:25.085698    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c5bca5-53b9-475e-a4d3-c7fa153c4c35" path="/var/lib/kubelet/pods/25c5bca5-53b9-475e-a4d3-c7fa153c4c35/volumes"
Nov 03 13:31:25.086203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:25.086189    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de14b3f-b00e-4105-b5e9-4b52ef946958" path="/var/lib/kubelet/pods/3de14b3f-b00e-4105-b5e9-4b52ef946958/volumes"
Nov 03 13:31:25.104296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:25.104276    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:25.388660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:25.388626    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"]
Nov 03 13:31:25.389069 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:31:25.389039    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1420976_939e_4a5a_96db_c41d7c234201.slice/crio-2e0d02fab1fc9391c47b9e4e545d6de1e1a5509d488cbf252487bf150c447344 WatchSource:0}: Error finding container 2e0d02fab1fc9391c47b9e4e545d6de1e1a5509d488cbf252487bf150c447344: Status 404 returned error can't find the container with id 2e0d02fab1fc9391c47b9e4e545d6de1e1a5509d488cbf252487bf150c447344
Nov 03 13:31:25.550979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:25.550943    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l" event={"ID":"b1420976-939e-4a5a-96db-c41d7c234201","Type":"ContainerStarted","Data":"2e0d02fab1fc9391c47b9e4e545d6de1e1a5509d488cbf252487bf150c447344"}
Nov 03 13:31:26.023243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:26.023176    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"]
Nov 03 13:31:26.027610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:26.027565    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669d50fc-cnlfb"]
Nov 03 13:31:26.562747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:26.562716    5420 generic.go:356] "Generic (PLEG): container finished" podID="b1420976-939e-4a5a-96db-c41d7c234201" containerID="d3115068baf650ee554ce4b46af63a38ef57a5f2c2427f9bffd441d98d3df330" exitCode=0
Nov 03 13:31:26.562936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:26.562805    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l" event={"ID":"b1420976-939e-4a5a-96db-c41d7c234201","Type":"ContainerDied","Data":"d3115068baf650ee554ce4b46af63a38ef57a5f2c2427f9bffd441d98d3df330"}
Nov 03 13:31:27.086409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.086377    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb086d0-ad43-449a-98f7-42b6fbc19195" path="/var/lib/kubelet/pods/beb086d0-ad43-449a-98f7-42b6fbc19195/volumes"
Nov 03 13:31:27.102376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.102347    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:27.235665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.235636    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:27.369550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.369473    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:27.514990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.514955    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:27.765964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.765942    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:27.916944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.916874    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-xks-ca\") pod \"b1420976-939e-4a5a-96db-c41d7c234201\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") "
Nov 03 13:31:27.916944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.916907    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1420976-939e-4a5a-96db-c41d7c234201-kube-api-access-m5xsg\" (UniqueName: \"kubernetes.io/projected/b1420976-939e-4a5a-96db-c41d7c234201-kube-api-access-m5xsg\") pod \"b1420976-939e-4a5a-96db-c41d7c234201\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") "
Nov 03 13:31:27.917134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.917006    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-openshift-ca\") pod \"b1420976-939e-4a5a-96db-c41d7c234201\" (UID: \"b1420976-939e-4a5a-96db-c41d7c234201\") "
Nov 03 13:31:27.917463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.917426    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-openshift-ca" pod "b1420976-939e-4a5a-96db-c41d7c234201" (UID: "b1420976-939e-4a5a-96db-c41d7c234201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:27.917591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.917568    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:27.919048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.919012    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-xks-ca" pod "b1420976-939e-4a5a-96db-c41d7c234201" (UID: "b1420976-939e-4a5a-96db-c41d7c234201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:31:27.919048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:27.919015    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1420976-939e-4a5a-96db-c41d7c234201-kube-api-access-m5xsg" pod "b1420976-939e-4a5a-96db-c41d7c234201" (UID: "b1420976-939e-4a5a-96db-c41d7c234201"). InnerVolumeSpecName "kube-api-access-m5xsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:28.019522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:28.019494    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1420976-939e-4a5a-96db-c41d7c234201-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:28.019522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:28.019519    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5xsg\" (UniqueName: \"kubernetes.io/projected/b1420976-939e-4a5a-96db-c41d7c234201-kube-api-access-m5xsg\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:28.124771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:28.122168    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:28.581431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:28.581405    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"
Nov 03 13:31:28.581431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:28.581427    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l" event={"ID":"b1420976-939e-4a5a-96db-c41d7c234201","Type":"ContainerDied","Data":"2e0d02fab1fc9391c47b9e4e545d6de1e1a5509d488cbf252487bf150c447344"}
Nov 03 13:31:28.581705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:28.581467    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e0d02fab1fc9391c47b9e4e545d6de1e1a5509d488cbf252487bf150c447344"
Nov 03 13:31:28.849608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:28.849510    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:29.523784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:29.523750    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:30.338802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:30.338773    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:30.670241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:30.670132    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:31:31.137520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:31.137489    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:31.976773 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:31.976737    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:32.999004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:32.998935    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:33.940237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:33.939468    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:34.918606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:34.918577    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:35.607552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:35.607505    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:36.261823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.261790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:36.470169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.470134    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:36.551349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.551268    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"]
Nov 03 13:31:36.552673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.552645    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b1420976-939e-4a5a-96db-c41d7c234201" containerName="revoke-keys"
Nov 03 13:31:36.552850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.552836    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1420976-939e-4a5a-96db-c41d7c234201" containerName="revoke-keys"
Nov 03 13:31:36.553160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.553145    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1420976-939e-4a5a-96db-c41d7c234201" containerName="revoke-keys"
Nov 03 13:31:36.588900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.587742    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"]
Nov 03 13:31:36.588900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.587917    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.600037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.600009    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-de83083e156c\""
Nov 03 13:31:36.600189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.600123    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1a18c8\""
Nov 03 13:31:36.600269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.600259    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-e2e--d832c86e-dockercfg-sq45c\""
Nov 03 13:31:36.600339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.600324    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-d522b5d8feb5\""
Nov 03 13:31:36.600423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.600411    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"routing-overlay\""
Nov 03 13:31:36.600501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.600489    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"kube-root-ca.crt\""
Nov 03 13:31:36.600560 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.600549    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-31f9f21f7966\""
Nov 03 13:31:36.601354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.601331    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:36.601475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.601461    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"openshift-service-ca.crt\""
Nov 03 13:31:36.642620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.642589    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.642780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.642723    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m4v4\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-kube-api-access-4m4v4\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.642913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.642896    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.642960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.642946    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.744105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.744071    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m4v4\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-kube-api-access-4m4v4\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.744323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.744267    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.744401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.744321    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.744459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.744432    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.744991 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.744968    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.745096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.745017    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.746899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.746876    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.752303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.752280    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m4v4\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-kube-api-access-4m4v4\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:36.928571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:36.928481    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:37.267940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.267805    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:37.355275 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:31:37.353386    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4924b03_ee1d_45c7_87d9_62b173bbb0b0.slice/crio-148666d97178e68a8c2e5a4b81c5254c5c9ff46675a3ef4b09e351fc57d8a73b WatchSource:0}: Error finding container 148666d97178e68a8c2e5a4b81c5254c5c9ff46675a3ef4b09e351fc57d8a73b: Status 404 returned error can't find the container with id 148666d97178e68a8c2e5a4b81c5254c5c9ff46675a3ef4b09e351fc57d8a73b
Nov 03 13:31:37.426056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.422328    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"]
Nov 03 13:31:37.451631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.451585    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:37.603779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.603743    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"]
Nov 03 13:31:37.608538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.608503    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.624245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.620066    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"]
Nov 03 13:31:37.624245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.621370    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ef1693\""
Nov 03 13:31:37.624245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.621448    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"openshift-service-ca.crt\""
Nov 03 13:31:37.624245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.622556    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-4d65870da959\""
Nov 03 13:31:37.624245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.622644    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"kube-root-ca.crt\""
Nov 03 13:31:37.624245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.622714    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"routing-overlay\""
Nov 03 13:31:37.624245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.622778    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-5e1bd9e90045\""
Nov 03 13:31:37.624245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.622833    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-1c2ab57ceeb2\""
Nov 03 13:31:37.624245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.622932    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-e2e--393fd9ed-dockercfg-fnc52\""
Nov 03 13:31:37.673243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.672136    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnwz\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-kube-api-access-djnwz\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.673243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.672239    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.673243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.672457    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.673243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.672557    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.722936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.722907    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9" event={"ID":"b4924b03-ee1d-45c7-87d9-62b173bbb0b0","Type":"ContainerStarted","Data":"428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c"}
Nov 03 13:31:37.723125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.722954    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9" event={"ID":"b4924b03-ee1d-45c7-87d9-62b173bbb0b0","Type":"ContainerStarted","Data":"148666d97178e68a8c2e5a4b81c5254c5c9ff46675a3ef4b09e351fc57d8a73b"}
Nov 03 13:31:37.723174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.723137    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:37.774283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.774249    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.774447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.774339    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.774447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.774429    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnwz\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-kube-api-access-djnwz\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.774570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.774482    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.775180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.775154    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.775317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.775285    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.776965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.776947    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.782979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.782926    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnwz\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-kube-api-access-djnwz\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:37.992198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:37.992166    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:38.135562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:38.135395    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:38.275475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:38.275433    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9" podStartSLOduration=2.27541655 podStartE2EDuration="2.27541655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:37.750434358 +0000 UTC m=+1338.828199386" watchObservedRunningTime="2026-11-03 13:31:38.27541655 +0000 UTC m=+1339.353181615"
Nov 03 13:31:38.276111 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:31:38.276087    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390a1dfc_eac5_4874_ad99_365b63770632.slice/crio-06f58ebd4e266d28dba7faaa4a880964a5c22cb96c0b4167e8fe4997adbe1234 WatchSource:0}: Error finding container 06f58ebd4e266d28dba7faaa4a880964a5c22cb96c0b4167e8fe4997adbe1234: Status 404 returned error can't find the container with id 06f58ebd4e266d28dba7faaa4a880964a5c22cb96c0b4167e8fe4997adbe1234
Nov 03 13:31:38.276529 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:38.276510    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"]
Nov 03 13:31:38.785884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:38.785839    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" event={"ID":"390a1dfc-eac5-4874-ad99-365b63770632","Type":"ContainerStarted","Data":"e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32"}
Nov 03 13:31:38.786352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:38.786324    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" event={"ID":"390a1dfc-eac5-4874-ad99-365b63770632","Type":"ContainerStarted","Data":"06f58ebd4e266d28dba7faaa4a880964a5c22cb96c0b4167e8fe4997adbe1234"}
Nov 03 13:31:38.786562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:38.786548    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:38.807272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:38.807228    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" podStartSLOduration=1.807203106 podStartE2EDuration="1.807203106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:38.805196929 +0000 UTC m=+1339.882961956" watchObservedRunningTime="2026-11-03 13:31:38.807203106 +0000 UTC m=+1339.884968147"
Nov 03 13:31:39.077787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:39.077650    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:39.447555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:39.447471    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"]
Nov 03 13:31:39.710518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:39.710482    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:39.905237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:39.896471    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:39.968292 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:39.955308    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ef1693/agc-cred-1c2ab57ceeb2: secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:39.968292 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:39.955343    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp: secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:39.968292 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:39.955410    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials podName:390a1dfc-eac5-4874-ad99-365b63770632 nodeName:}" failed. No retries permitted until 2026-11-03 13:31:40.455390604 +0000 UTC m=+1341.533155629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" (UID: "390a1dfc-eac5-4874-ad99-365b63770632") : secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:40.282789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.282702    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"]
Nov 03 13:31:40.287166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.287134    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.293062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.293035    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:31:40.293557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.293537    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:31:40.294680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.294662    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:31:40.294680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.294673    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:31:40.296359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.296341    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"]
Nov 03 13:31:40.368887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.368859    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zbf\" (UniqueName: \"kubernetes.io/projected/b2940d67-1ee4-40af-ad03-195e9da36635-kube-api-access-m8zbf\") pod \"maas-api-revoke-sub-fcf68801-92m8l\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.369063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.369035    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf68801-92m8l\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.369107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.369079    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf68801-92m8l\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.443953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.443892    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:40.471815 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.471766    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zbf\" (UniqueName: \"kubernetes.io/projected/b2940d67-1ee4-40af-ad03-195e9da36635-kube-api-access-m8zbf\") pod \"maas-api-revoke-sub-fcf68801-92m8l\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.472360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.471989    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf68801-92m8l\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.472360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.472045    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf68801-92m8l\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.472360 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:40.472084    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ef1693/agc-cred-1c2ab57ceeb2: secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:40.472360 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:40.472113    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp: secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:40.472360 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:40.472168    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials podName:390a1dfc-eac5-4874-ad99-365b63770632 nodeName:}" failed. No retries permitted until 2026-11-03 13:31:41.472148745 +0000 UTC m=+1342.549913770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" (UID: "390a1dfc-eac5-4874-ad99-365b63770632") : secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:40.473827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.473807    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf68801-92m8l\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.474990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.474970    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf68801-92m8l\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.484617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.484585    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zbf\" (UniqueName: \"kubernetes.io/projected/b2940d67-1ee4-40af-ad03-195e9da36635-kube-api-access-m8zbf\") pod \"maas-api-revoke-sub-fcf68801-92m8l\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.604488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.604409    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:40.685857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.685827    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"]
Nov 03 13:31:40.891475 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:31:40.891443    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2940d67_1ee4_40af_ad03_195e9da36635.slice/crio-81477244ce5530789cce7961600eadf93ce8cc59a6951e9a0cb691a897cbb28e WatchSource:0}: Error finding container 81477244ce5530789cce7961600eadf93ce8cc59a6951e9a0cb691a897cbb28e: Status 404 returned error can't find the container with id 81477244ce5530789cce7961600eadf93ce8cc59a6951e9a0cb691a897cbb28e
Nov 03 13:31:40.891608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.891485    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"]
Nov 03 13:31:40.908088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:40.908064    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l" event={"ID":"b2940d67-1ee4-40af-ad03-195e9da36635","Type":"ContainerStarted","Data":"81477244ce5530789cce7961600eadf93ce8cc59a6951e9a0cb691a897cbb28e"}
Nov 03 13:31:41.160541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:41.160486    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:41.510235 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:41.496198    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ef1693/agc-cred-1c2ab57ceeb2: secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:41.510235 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:41.496253    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp: secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:41.510235 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:41.496360    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials podName:390a1dfc-eac5-4874-ad99-365b63770632 nodeName:}" failed. No retries permitted until 2026-11-03 13:31:43.496335117 +0000 UTC m=+1344.574100135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" (UID: "390a1dfc-eac5-4874-ad99-365b63770632") : secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:41.801682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:41.798331    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:41.920611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:41.920572    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l" event={"ID":"b2940d67-1ee4-40af-ad03-195e9da36635","Type":"ContainerStarted","Data":"45d44b2b8ab985b90280afb62ddbffccb7d14a2b05168941da23612fce703189"}
Nov 03 13:31:41.945696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:41.945650    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l" podStartSLOduration=1.9456356879999999 podStartE2EDuration="1.945635688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:41.942557984 +0000 UTC m=+1343.020323010" watchObservedRunningTime="2026-11-03 13:31:41.945635688 +0000 UTC m=+1343.023400715"
Nov 03 13:31:42.421150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:42.421094    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:42.931376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:42.931342    5420 generic.go:356] "Generic (PLEG): container finished" podID="b2940d67-1ee4-40af-ad03-195e9da36635" containerID="45d44b2b8ab985b90280afb62ddbffccb7d14a2b05168941da23612fce703189" exitCode=0
Nov 03 13:31:42.931795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:42.931425    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l" event={"ID":"b2940d67-1ee4-40af-ad03-195e9da36635","Type":"ContainerDied","Data":"45d44b2b8ab985b90280afb62ddbffccb7d14a2b05168941da23612fce703189"}
Nov 03 13:31:43.017573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:43.017541    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:43.538777 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:43.538745    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ef1693/agc-cred-1c2ab57ceeb2: secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:43.538777 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:43.538776    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp: secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:43.538977 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:43.538826    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials podName:390a1dfc-eac5-4874-ad99-365b63770632 nodeName:}" failed. No retries permitted until 2026-11-03 13:31:47.538811473 +0000 UTC m=+1348.616576481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" (UID: "390a1dfc-eac5-4874-ad99-365b63770632") : secret "agc-cred-1c2ab57ceeb2" not found
Nov 03 13:31:43.630242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:43.627046    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:44.125547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.125508    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:44.222167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.222132    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:44.251428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.251397    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2940d67-1ee4-40af-ad03-195e9da36635-kube-api-access-m8zbf\" (UniqueName: \"kubernetes.io/projected/b2940d67-1ee4-40af-ad03-195e9da36635-kube-api-access-m8zbf\") pod \"b2940d67-1ee4-40af-ad03-195e9da36635\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") "
Nov 03 13:31:44.251703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.251498    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-xks-ca\") pod \"b2940d67-1ee4-40af-ad03-195e9da36635\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") "
Nov 03 13:31:44.251703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.251682    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-openshift-ca\") pod \"b2940d67-1ee4-40af-ad03-195e9da36635\" (UID: \"b2940d67-1ee4-40af-ad03-195e9da36635\") "
Nov 03 13:31:44.251997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.251956    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-openshift-ca" pod "b2940d67-1ee4-40af-ad03-195e9da36635" (UID: "b2940d67-1ee4-40af-ad03-195e9da36635"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:44.252680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.252656    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:44.253859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.253825    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2940d67-1ee4-40af-ad03-195e9da36635-kube-api-access-m8zbf" pod "b2940d67-1ee4-40af-ad03-195e9da36635" (UID: "b2940d67-1ee4-40af-ad03-195e9da36635"). InnerVolumeSpecName "kube-api-access-m8zbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:44.254636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.254617    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-xks-ca" pod "b2940d67-1ee4-40af-ad03-195e9da36635" (UID: "b2940d67-1ee4-40af-ad03-195e9da36635"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:31:44.354865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.354832    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zbf\" (UniqueName: \"kubernetes.io/projected/b2940d67-1ee4-40af-ad03-195e9da36635-kube-api-access-m8zbf\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:44.354865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.354863    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2940d67-1ee4-40af-ad03-195e9da36635-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:44.898892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.898839    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9" podUID="b4924b03-ee1d-45c7-87d9-62b173bbb0b0" containerName="payload-processing" containerID="cri-o://428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c" gracePeriod=25
Nov 03 13:31:44.957547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.957516    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:44.984862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.984835    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"
Nov 03 13:31:44.984862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.984850    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l" event={"ID":"b2940d67-1ee4-40af-ad03-195e9da36635","Type":"ContainerDied","Data":"81477244ce5530789cce7961600eadf93ce8cc59a6951e9a0cb691a897cbb28e"}
Nov 03 13:31:44.985051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:44.984903    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81477244ce5530789cce7961600eadf93ce8cc59a6951e9a0cb691a897cbb28e"
Nov 03 13:31:45.201368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.201342    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:45.369642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.369611    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-kube-api-access-4m4v4\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-kube-api-access-4m4v4\") pod \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") "
Nov 03 13:31:45.369846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.369742    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-provider-credentials\") pod \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") "
Nov 03 13:31:45.369846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.369818    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-plugins-config-volume\") pod \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") "
Nov 03 13:31:45.370012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.369994    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-routing-overlay-volume\") pod \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\" (UID: \"b4924b03-ee1d-45c7-87d9-62b173bbb0b0\") "
Nov 03 13:31:45.370285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.370259    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-plugins-config-volume" pod "b4924b03-ee1d-45c7-87d9-62b173bbb0b0" (UID: "b4924b03-ee1d-45c7-87d9-62b173bbb0b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:45.370492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.370452    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-routing-overlay-volume" pod "b4924b03-ee1d-45c7-87d9-62b173bbb0b0" (UID: "b4924b03-ee1d-45c7-87d9-62b173bbb0b0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:45.370905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.370885    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:45.370963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.370911    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:45.372046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.372028    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-provider-credentials" pod "b4924b03-ee1d-45c7-87d9-62b173bbb0b0" (UID: "b4924b03-ee1d-45c7-87d9-62b173bbb0b0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:45.372126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.372067    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-kube-api-access-4m4v4" pod "b4924b03-ee1d-45c7-87d9-62b173bbb0b0" (UID: "b4924b03-ee1d-45c7-87d9-62b173bbb0b0"). InnerVolumeSpecName "kube-api-access-4m4v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:45.472448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.472423    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:45.472448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.472448    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m4v4\" (UniqueName: \"kubernetes.io/projected/b4924b03-ee1d-45c7-87d9-62b173bbb0b0-kube-api-access-4m4v4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:45.620741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.620711    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:45.909283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.909176    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" podUID="390a1dfc-eac5-4874-ad99-365b63770632" containerName="payload-processing" containerID="cri-o://e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32" gracePeriod=25
Nov 03 13:31:45.994952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.994918    5420 generic.go:356] "Generic (PLEG): container finished" podID="b4924b03-ee1d-45c7-87d9-62b173bbb0b0" containerID="428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c" exitCode=0
Nov 03 13:31:45.995101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.994970    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"
Nov 03 13:31:45.995101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.995005    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9" event={"ID":"b4924b03-ee1d-45c7-87d9-62b173bbb0b0","Type":"ContainerDied","Data":"428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c"}
Nov 03 13:31:45.995101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.995064    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9" event={"ID":"b4924b03-ee1d-45c7-87d9-62b173bbb0b0","Type":"ContainerDied","Data":"148666d97178e68a8c2e5a4b81c5254c5c9ff46675a3ef4b09e351fc57d8a73b"}
Nov 03 13:31:45.995101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:45.995093    5420 scope.go:122] "RemoveContainer" containerID="428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c"
Nov 03 13:31:46.024269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.023963    5420 scope.go:122] "RemoveContainer" containerID="428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c"
Nov 03 13:31:46.024454 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:46.024427    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c\": container with ID starting with 428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c not found: ID does not exist" containerID="428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c"
Nov 03 13:31:46.024552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.024468    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c"} err="failed to get container status \"428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c\": rpc error: code = NotFound desc = could not find container \"428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c\": container with ID starting with 428b724866c8e3618330b3ed0f318e6bc7396639129f367cd2df67fd99a5de7c not found: ID does not exist"
Nov 03 13:31:46.024667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.024624    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"]
Nov 03 13:31:46.030376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.030354    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"]
Nov 03 13:31:46.035298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.035275    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-c6ddshpj9"]
Nov 03 13:31:46.039625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.039602    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3aa55a0-b46lj"]
Nov 03 13:31:46.223816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.222813    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:46.233715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.233483    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:46.386316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.386285    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-routing-overlay-volume\") pod \"390a1dfc-eac5-4874-ad99-365b63770632\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") "
Nov 03 13:31:46.386510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.386406    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-plugins-config-volume\") pod \"390a1dfc-eac5-4874-ad99-365b63770632\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") "
Nov 03 13:31:46.386510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.386432    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials\") pod \"390a1dfc-eac5-4874-ad99-365b63770632\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") "
Nov 03 13:31:46.386510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.386502    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-kube-api-access-djnwz\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-kube-api-access-djnwz\") pod \"390a1dfc-eac5-4874-ad99-365b63770632\" (UID: \"390a1dfc-eac5-4874-ad99-365b63770632\") "
Nov 03 13:31:46.386693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.386629    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-routing-overlay-volume" pod "390a1dfc-eac5-4874-ad99-365b63770632" (UID: "390a1dfc-eac5-4874-ad99-365b63770632"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:46.386866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.386844    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-plugins-config-volume" pod "390a1dfc-eac5-4874-ad99-365b63770632" (UID: "390a1dfc-eac5-4874-ad99-365b63770632"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:46.387357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.387336    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:46.387445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.387364    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390a1dfc-eac5-4874-ad99-365b63770632-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:46.388708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.388680    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-kube-api-access-djnwz" pod "390a1dfc-eac5-4874-ad99-365b63770632" (UID: "390a1dfc-eac5-4874-ad99-365b63770632"). InnerVolumeSpecName "kube-api-access-djnwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:46.388839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.388816    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials" pod "390a1dfc-eac5-4874-ad99-365b63770632" (UID: "390a1dfc-eac5-4874-ad99-365b63770632"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:46.489495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.489458    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:46.489495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.489489    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djnwz\" (UniqueName: \"kubernetes.io/projected/390a1dfc-eac5-4874-ad99-365b63770632-kube-api-access-djnwz\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:46.849986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:46.849888    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:47.004860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.004828    5420 generic.go:356] "Generic (PLEG): container finished" podID="390a1dfc-eac5-4874-ad99-365b63770632" containerID="e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32" exitCode=0
Nov 03 13:31:47.005052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.004893    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"
Nov 03 13:31:47.005052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.004903    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" event={"ID":"390a1dfc-eac5-4874-ad99-365b63770632","Type":"ContainerDied","Data":"e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32"}
Nov 03 13:31:47.005052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.004988    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp" event={"ID":"390a1dfc-eac5-4874-ad99-365b63770632","Type":"ContainerDied","Data":"06f58ebd4e266d28dba7faaa4a880964a5c22cb96c0b4167e8fe4997adbe1234"}
Nov 03 13:31:47.005052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.005019    5420 scope.go:122] "RemoveContainer" containerID="e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32"
Nov 03 13:31:47.016948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.016930    5420 scope.go:122] "RemoveContainer" containerID="e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32"
Nov 03 13:31:47.017231 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:47.017193    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32\": container with ID starting with e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32 not found: ID does not exist" containerID="e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32"
Nov 03 13:31:47.017278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.017235    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32"} err="failed to get container status \"e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32\": rpc error: code = NotFound desc = could not find container \"e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32\": container with ID starting with e460bbf174eb898a09db450c847c0d4da03f0be988eadc979b129ac138e09d32 not found: ID does not exist"
Nov 03 13:31:47.034720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.034694    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"]
Nov 03 13:31:47.039109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.039088    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66fzgwp"]
Nov 03 13:31:47.088189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.088160    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15437773-5d9d-4743-ae1b-5b84f43c1daf" path="/var/lib/kubelet/pods/15437773-5d9d-4743-ae1b-5b84f43c1daf/volumes"
Nov 03 13:31:47.088678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.088662    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390a1dfc-eac5-4874-ad99-365b63770632" path="/var/lib/kubelet/pods/390a1dfc-eac5-4874-ad99-365b63770632/volumes"
Nov 03 13:31:47.089056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.089044    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4924b03-ee1d-45c7-87d9-62b173bbb0b0" path="/var/lib/kubelet/pods/b4924b03-ee1d-45c7-87d9-62b173bbb0b0/volumes"
Nov 03 13:31:47.650779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:47.650682    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:48.156785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:48.151734    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:48.737230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:48.737109    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:49.385920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:49.385835    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:50.226388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.226310    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:50.799804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.799772    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"]
Nov 03 13:31:50.800925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.800660    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b4924b03-ee1d-45c7-87d9-62b173bbb0b0" containerName="payload-processing"
Nov 03 13:31:50.800925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.800684    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4924b03-ee1d-45c7-87d9-62b173bbb0b0" containerName="payload-processing"
Nov 03 13:31:50.800925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.800742    5420 cpu_manager.go:414] "containerMap: removing container" podUID="390a1dfc-eac5-4874-ad99-365b63770632" containerName="payload-processing"
Nov 03 13:31:50.800925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.800750    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390a1dfc-eac5-4874-ad99-365b63770632" containerName="payload-processing"
Nov 03 13:31:50.800925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.800773    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b2940d67-1ee4-40af-ad03-195e9da36635" containerName="revoke-keys"
Nov 03 13:31:50.800925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.800781    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2940d67-1ee4-40af-ad03-195e9da36635" containerName="revoke-keys"
Nov 03 13:31:50.801396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.800948    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4924b03-ee1d-45c7-87d9-62b173bbb0b0" containerName="payload-processing"
Nov 03 13:31:50.801396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.800970    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="390a1dfc-eac5-4874-ad99-365b63770632" containerName="payload-processing"
Nov 03 13:31:50.801396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.800983    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2940d67-1ee4-40af-ad03-195e9da36635" containerName="revoke-keys"
Nov 03 13:31:50.819852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.818766    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:50.832063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.825746    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"]
Nov 03 13:31:50.836622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.832615    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:31:50.836622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.832801    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:31:50.836622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.832906    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:31:50.836622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.835357    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:31:50.848487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.848459    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:50.876488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.873043    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75503be4-ea90-45be-9925-3632c0853539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba4ccb5d-wrwwc\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:50.876845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.876818    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75503be4-ea90-45be-9925-3632c0853539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba4ccb5d-wrwwc\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:50.877094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.877071    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhx7h\" (UniqueName: \"kubernetes.io/projected/75503be4-ea90-45be-9925-3632c0853539-kube-api-access-jhx7h\") pod \"maas-api-revoke-sub-ba4ccb5d-wrwwc\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:50.978928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.978897    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75503be4-ea90-45be-9925-3632c0853539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba4ccb5d-wrwwc\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:50.979118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.979007    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75503be4-ea90-45be-9925-3632c0853539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba4ccb5d-wrwwc\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:50.979163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.979115    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhx7h\" (UniqueName: \"kubernetes.io/projected/75503be4-ea90-45be-9925-3632c0853539-kube-api-access-jhx7h\") pod \"maas-api-revoke-sub-ba4ccb5d-wrwwc\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:50.979640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.979617    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75503be4-ea90-45be-9925-3632c0853539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba4ccb5d-wrwwc\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:50.981292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.981276    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75503be4-ea90-45be-9925-3632c0853539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba4ccb5d-wrwwc\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:50.988634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:50.988611    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhx7h\" (UniqueName: \"kubernetes.io/projected/75503be4-ea90-45be-9925-3632c0853539-kube-api-access-jhx7h\") pod \"maas-api-revoke-sub-ba4ccb5d-wrwwc\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:51.179689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:51.179616    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:51.427639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:51.426578    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:51.467450 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:31:51.467420    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75503be4_ea90_45be_9925_3632c0853539.slice/crio-d010c127403da759bec5f7588bbe910184bb61ba7bdd3aca08caa3baa13f66ee WatchSource:0}: Error finding container d010c127403da759bec5f7588bbe910184bb61ba7bdd3aca08caa3baa13f66ee: Status 404 returned error can't find the container with id d010c127403da759bec5f7588bbe910184bb61ba7bdd3aca08caa3baa13f66ee
Nov 03 13:31:51.469895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:51.469868    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"]
Nov 03 13:31:52.003776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.003694    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:52.056513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.056481    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc" event={"ID":"75503be4-ea90-45be-9925-3632c0853539","Type":"ContainerStarted","Data":"308d8b53f17d38499c5c86167824a4936dd3d7f55ae105fb481d7bf32935c602"}
Nov 03 13:31:52.056691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.056540    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc" event={"ID":"75503be4-ea90-45be-9925-3632c0853539","Type":"ContainerStarted","Data":"d010c127403da759bec5f7588bbe910184bb61ba7bdd3aca08caa3baa13f66ee"}
Nov 03 13:31:52.082261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.078851    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc" podStartSLOduration=2.078833659 podStartE2EDuration="2.078833659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:52.076108504 +0000 UTC m=+1353.153873531" watchObservedRunningTime="2026-11-03 13:31:52.078833659 +0000 UTC m=+1353.156598688"
Nov 03 13:31:52.639174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.628279    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:52.724101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.724064    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"]
Nov 03 13:31:52.731179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.730319    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.735428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.735403    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-08111b-openshift-default-dockercfg-j69cw\""
Nov 03 13:31:52.736336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.736308    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"]
Nov 03 13:31:52.821316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.821284    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-podinfo\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.821471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.821413    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcc2d\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-kube-api-access-kcc2d\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.821471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.821457    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.821560 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.821547    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-credential-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.821607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.821595    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-token\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.821652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.821637    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-certs\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.821727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.821708    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f84e6b-bd58-4d15-a856-2900abe93ec4-istiod-ca-cert\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.821791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.821773    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-envoy\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.821830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.821815    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-data\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.923820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.923749    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-podinfo\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.923956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.923843    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcc2d\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-kube-api-access-kcc2d\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.923956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.923893    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.923956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.923950    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-credential-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924010    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-token\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924150    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-certs\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924196    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f84e6b-bd58-4d15-a856-2900abe93ec4-istiod-ca-cert\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924261    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-envoy\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924308    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-data\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924349    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924411    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-credential-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924466    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-certs\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924585    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-data\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.924952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.924928    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f84e6b-bd58-4d15-a856-2900abe93ec4-istiod-ca-cert\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.926323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.926303    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-podinfo\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.926419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.926359    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-envoy\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.933252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.933223    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-token\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:52.933414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:52.933391    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcc2d\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-kube-api-access-kcc2d\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:53.049088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:53.049055    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:53.068088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:53.068052    5420 generic.go:356] "Generic (PLEG): container finished" podID="75503be4-ea90-45be-9925-3632c0853539" containerID="308d8b53f17d38499c5c86167824a4936dd3d7f55ae105fb481d7bf32935c602" exitCode=0
Nov 03 13:31:53.068247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:53.068138    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc" event={"ID":"75503be4-ea90-45be-9925-3632c0853539","Type":"ContainerDied","Data":"308d8b53f17d38499c5c86167824a4936dd3d7f55ae105fb481d7bf32935c602"}
Nov 03 13:31:53.351316 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:31:53.351275    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f84e6b_bd58_4d15_a856_2900abe93ec4.slice/crio-b95d261f1db029214b76baae8c2d7f851aa77155a78db76b42076bb507e85e3b WatchSource:0}: Error finding container b95d261f1db029214b76baae8c2d7f851aa77155a78db76b42076bb507e85e3b: Status 404 returned error can't find the container with id b95d261f1db029214b76baae8c2d7f851aa77155a78db76b42076bb507e85e3b
Nov 03 13:31:53.353062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:53.353035    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"]
Nov 03 13:31:53.354570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:53.354527    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:31:53.354677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:53.354617    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:31:53.354677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:53.354661    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:31:53.430361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:53.430328    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:54.078488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:54.078453    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" event={"ID":"17f84e6b-bd58-4d15-a856-2900abe93ec4","Type":"ContainerStarted","Data":"f8cf5dd9168b8af2fc0a1e1c4f65d33a4c5c474d12be7d328f5a43e31be4083d"}
Nov 03 13:31:54.078960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:54.078509    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" event={"ID":"17f84e6b-bd58-4d15-a856-2900abe93ec4","Type":"ContainerStarted","Data":"b95d261f1db029214b76baae8c2d7f851aa77155a78db76b42076bb507e85e3b"}
Nov 03 13:31:54.177652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:54.174043    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" podStartSLOduration=2.174023589 podStartE2EDuration="2.174023589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:54.122103594 +0000 UTC m=+1355.199868621" watchObservedRunningTime="2026-11-03 13:31:54.174023589 +0000 UTC m=+1355.251788616"
Nov 03 13:31:54.700231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:54.699673    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:54.966523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:54.938758    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:55.007531 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:31:55.007500    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:31:55.059234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.056960    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:31:55.059234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.057089    5420 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 13:31:55.059234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.057146    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 13:31:55.098233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.097936    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75503be4-ea90-45be-9925-3632c0853539-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75503be4-ea90-45be-9925-3632c0853539-maas-api-xks-ca\") pod \"75503be4-ea90-45be-9925-3632c0853539\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") "
Nov 03 13:31:55.098233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.098190    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75503be4-ea90-45be-9925-3632c0853539-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75503be4-ea90-45be-9925-3632c0853539-maas-api-openshift-ca\") pod \"75503be4-ea90-45be-9925-3632c0853539\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") "
Nov 03 13:31:55.098793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.098265    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75503be4-ea90-45be-9925-3632c0853539-kube-api-access-jhx7h\" (UniqueName: \"kubernetes.io/projected/75503be4-ea90-45be-9925-3632c0853539-kube-api-access-jhx7h\") pod \"75503be4-ea90-45be-9925-3632c0853539\" (UID: \"75503be4-ea90-45be-9925-3632c0853539\") "
Nov 03 13:31:55.118311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.102544    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75503be4-ea90-45be-9925-3632c0853539-maas-api-xks-ca" pod "75503be4-ea90-45be-9925-3632c0853539" (UID: "75503be4-ea90-45be-9925-3632c0853539"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:31:55.118311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.102869    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75503be4-ea90-45be-9925-3632c0853539-maas-api-openshift-ca" pod "75503be4-ea90-45be-9925-3632c0853539" (UID: "75503be4-ea90-45be-9925-3632c0853539"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:31:55.160292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.144424    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"
Nov 03 13:31:55.187236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.186644    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75503be4-ea90-45be-9925-3632c0853539-kube-api-access-jhx7h" pod "75503be4-ea90-45be-9925-3632c0853539" (UID: "75503be4-ea90-45be-9925-3632c0853539"). InnerVolumeSpecName "kube-api-access-jhx7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:31:55.201267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.200192    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc" event={"ID":"75503be4-ea90-45be-9925-3632c0853539","Type":"ContainerDied","Data":"d010c127403da759bec5f7588bbe910184bb61ba7bdd3aca08caa3baa13f66ee"}
Nov 03 13:31:55.201267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.200286    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d010c127403da759bec5f7588bbe910184bb61ba7bdd3aca08caa3baa13f66ee"
Nov 03 13:31:55.208473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.207938    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75503be4-ea90-45be-9925-3632c0853539-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:55.208473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.207976    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75503be4-ea90-45be-9925-3632c0853539-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:55.208473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.207991    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhx7h\" (UniqueName: \"kubernetes.io/projected/75503be4-ea90-45be-9925-3632c0853539-kube-api-access-jhx7h\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:31:55.394282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:55.393697    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:56.053237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:56.050582    5420 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 13:31:56.053237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:56.050640    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 13:31:56.233237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:56.220742    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:57.061556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:57.051632    5420 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 13:31:57.061556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:57.051686    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 13:31:57.123235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:57.110457    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"]
Nov 03 13:31:57.177232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:57.167060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:57.250236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:57.250058    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-380ac2d7-twj6q"]
Nov 03 13:31:57.874236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:57.869846    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:31:57.974235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:57.971958    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:58.061237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:58.061053    5420 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 13:31:58.061237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:58.061107    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 13:31:58.694232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:58.687221    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:59.050239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:59.049838    5420 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 13:31:59.050239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:59.049916    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 13:31:59.086236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:59.086191    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be070a20-aa77-448a-a06d-93c631c036f3" path="/var/lib/kubelet/pods/be070a20-aa77-448a-a06d-93c631c036f3/volumes"
Nov 03 13:31:59.342799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:31:59.342678    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:00.053089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:00.052642    5420 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 13:32:00.053089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:00.052706    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 13:32:00.063735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:00.063684    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:00.727902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:00.727818    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:01.095053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:01.094975    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:32:01.096295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:01.096059    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:32:01.097142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:01.097095    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:32:01.744833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:01.744796    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:02.060859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.060781    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"]
Nov 03 13:32:02.061561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.061545    5420 cpu_manager.go:414] "containerMap: removing container" podUID="75503be4-ea90-45be-9925-3632c0853539" containerName="revoke-keys"
Nov 03 13:32:02.061616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.061562    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75503be4-ea90-45be-9925-3632c0853539" containerName="revoke-keys"
Nov 03 13:32:02.061704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.061695    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="75503be4-ea90-45be-9925-3632c0853539" containerName="revoke-keys"
Nov 03 13:32:02.066529 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.066512    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.074039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.074012    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3de2312914d9\""
Nov 03 13:32:02.076071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.075844    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 13:32:02.076071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.075926    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 13:32:02.076071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.076031    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vx42v\""
Nov 03 13:32:02.076351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.076178    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 13:32:02.076534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.076514    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 13:32:02.082165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.082144    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"]
Nov 03 13:32:02.206154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.206128    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-routing-overlay-volume\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.206599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.206183    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn7r4\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-kube-api-access-zn7r4\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.206599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.206358    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-plugins-config-volume\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.206599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.206553    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-provider-credentials\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.308390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.308354    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-plugins-config-volume\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.308559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.308504    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-provider-credentials\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.308666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.308651    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-routing-overlay-volume\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.308787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.308719    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7r4\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-kube-api-access-zn7r4\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.309277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.309197    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-plugins-config-volume\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.309382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.309278    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-routing-overlay-volume\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.310958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.310906    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-provider-credentials\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.326742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.326718    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7r4\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-kube-api-access-zn7r4\") pod \"payload-processing-external-model-69d95d887d-zpbks\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.383694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.383671    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:02.577650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.577541    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.720291    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"]
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.728689    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"]
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.736274    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"]
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.736383    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.742491    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"kube-root-ca.crt\""
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.743090    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-e2e--393fd9ed-dockercfg-fnc52\""
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.743172    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ef1693\""
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.743246    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"routing-overlay\""
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.743297    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-4d65870da959\""
Nov 03 13:32:02.744548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.743412    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"openshift-service-ca.crt\""
Nov 03 13:32:02.759909 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.759057    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:02.830500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.830071    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.830782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.830753    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.830945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.830928    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gpn2\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-kube-api-access-2gpn2\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.831081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.831065    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.932813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.932787    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.932920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.932848    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.932920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.932882    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gpn2\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-kube-api-access-2gpn2\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.933109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.933083    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.933726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.933701    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.933865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.933842    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.935791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.935768    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:02.941995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:02.941973    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gpn2\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-kube-api-access-2gpn2\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:03.128681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.128587    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:03.155370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.155334    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"]
Nov 03 13:32:03.175476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.175446    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.175709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.175684    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"]
Nov 03 13:32:03.184503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.184475    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:32:03.185162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.185138    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:32:03.185304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.185238    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:32:03.185374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.185305    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:32:03.348605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.348566    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks" event={"ID":"7dc71620-7536-4e26-ac8e-f0f28e2e21b3","Type":"ContainerStarted","Data":"76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1"}
Nov 03 13:32:03.349116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.348628    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks" event={"ID":"7dc71620-7536-4e26-ac8e-f0f28e2e21b3","Type":"ContainerStarted","Data":"aca2a203d7e5578178a40009ad87ed7fdcd7b215a72c25d81bdc364a6864d0f4"}
Nov 03 13:32:03.349116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.348659    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:03.386317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.372566    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpsk\" (UniqueName: \"kubernetes.io/projected/465ea315-d67b-499d-828b-d020e913be4b-kube-api-access-8rpsk\") pod \"maas-api-revoke-sub-98bd6923-znkk8\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.386317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.372638    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ea315-d67b-499d-828b-d020e913be4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bd6923-znkk8\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.386317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.372844    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ea315-d67b-499d-828b-d020e913be4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bd6923-znkk8\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.455320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.455265    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks" podStartSLOduration=1.455246432 podStartE2EDuration="1.455246432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:03.390447286 +0000 UTC m=+1364.468212313" watchObservedRunningTime="2026-11-03 13:32:03.455246432 +0000 UTC m=+1364.533011454"
Nov 03 13:32:03.476784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.476725    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpsk\" (UniqueName: \"kubernetes.io/projected/465ea315-d67b-499d-828b-d020e913be4b-kube-api-access-8rpsk\") pod \"maas-api-revoke-sub-98bd6923-znkk8\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.476979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.476813    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ea315-d67b-499d-828b-d020e913be4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bd6923-znkk8\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.477059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.476995    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ea315-d67b-499d-828b-d020e913be4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bd6923-znkk8\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.478081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.478056    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ea315-d67b-499d-828b-d020e913be4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bd6923-znkk8\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.480341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.480314    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ea315-d67b-499d-828b-d020e913be4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bd6923-znkk8\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.488188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.488161    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"]
Nov 03 13:32:03.488408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.488383    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpsk\" (UniqueName: \"kubernetes.io/projected/465ea315-d67b-499d-828b-d020e913be4b-kube-api-access-8rpsk\") pod \"maas-api-revoke-sub-98bd6923-znkk8\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.522294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.516839    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:03.546550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.545144    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:03.718942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.718895    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:03.891961 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:32:03.891918    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465ea315_d67b_499d_828b_d020e913be4b.slice/crio-09c68ee044bbeca5e49416123df29fdfc44334e840c317e44fc6a8707dc076fb WatchSource:0}: Error finding container 09c68ee044bbeca5e49416123df29fdfc44334e840c317e44fc6a8707dc076fb: Status 404 returned error can't find the container with id 09c68ee044bbeca5e49416123df29fdfc44334e840c317e44fc6a8707dc076fb
Nov 03 13:32:03.892488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:03.892460    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"]
Nov 03 13:32:04.359360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:04.359333    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5" event={"ID":"04f900ce-bd07-4fc3-9ab8-456c069204b8","Type":"ContainerStarted","Data":"9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8"}
Nov 03 13:32:04.359780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:04.359391    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5" event={"ID":"04f900ce-bd07-4fc3-9ab8-456c069204b8","Type":"ContainerStarted","Data":"9a48209c129872a2d5a55519bf6d59525ffdbba224f9df2a5cd362f265e30cef"}
Nov 03 13:32:04.359780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:04.359440    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:04.361266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:04.361241    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8" event={"ID":"465ea315-d67b-499d-828b-d020e913be4b","Type":"ContainerStarted","Data":"aacbfc4e651dd7170f67e16d753af63d15e9b12d4839d1dd70acf178aebd01eb"}
Nov 03 13:32:04.361406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:04.361287    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8" event={"ID":"465ea315-d67b-499d-828b-d020e913be4b","Type":"ContainerStarted","Data":"09c68ee044bbeca5e49416123df29fdfc44334e840c317e44fc6a8707dc076fb"}
Nov 03 13:32:04.378763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:04.378708    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5" podStartSLOduration=2.378696003 podStartE2EDuration="2.378696003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:04.376059875 +0000 UTC m=+1365.453824902" watchObservedRunningTime="2026-11-03 13:32:04.378696003 +0000 UTC m=+1365.456461068"
Nov 03 13:32:04.400012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:04.399968    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8" podStartSLOduration=1.399957776 podStartE2EDuration="1.399957776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:04.397840342 +0000 UTC m=+1365.475605371" watchObservedRunningTime="2026-11-03 13:32:04.399957776 +0000 UTC m=+1365.477722802"
Nov 03 13:32:04.628700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:04.628572    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:05.391253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:05.391191    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:05.394889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:05.394866    5420 generic.go:356] "Generic (PLEG): container finished" podID="465ea315-d67b-499d-828b-d020e913be4b" containerID="aacbfc4e651dd7170f67e16d753af63d15e9b12d4839d1dd70acf178aebd01eb" exitCode=0
Nov 03 13:32:05.395086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:05.394956    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8" event={"ID":"465ea315-d67b-499d-828b-d020e913be4b","Type":"ContainerDied","Data":"aacbfc4e651dd7170f67e16d753af63d15e9b12d4839d1dd70acf178aebd01eb"}
Nov 03 13:32:06.299647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.298974    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:06.580184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.580154    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:06.648070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.648031    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465ea315-d67b-499d-828b-d020e913be4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ea315-d67b-499d-828b-d020e913be4b-maas-api-xks-ca\") pod \"465ea315-d67b-499d-828b-d020e913be4b\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") "
Nov 03 13:32:06.648292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.648100    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465ea315-d67b-499d-828b-d020e913be4b-kube-api-access-8rpsk\" (UniqueName: \"kubernetes.io/projected/465ea315-d67b-499d-828b-d020e913be4b-kube-api-access-8rpsk\") pod \"465ea315-d67b-499d-828b-d020e913be4b\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") "
Nov 03 13:32:06.648292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.648272    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465ea315-d67b-499d-828b-d020e913be4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ea315-d67b-499d-828b-d020e913be4b-maas-api-openshift-ca\") pod \"465ea315-d67b-499d-828b-d020e913be4b\" (UID: \"465ea315-d67b-499d-828b-d020e913be4b\") "
Nov 03 13:32:06.648661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.648634    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465ea315-d67b-499d-828b-d020e913be4b-maas-api-openshift-ca" pod "465ea315-d67b-499d-828b-d020e913be4b" (UID: "465ea315-d67b-499d-828b-d020e913be4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:06.649176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.649152    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ea315-d67b-499d-828b-d020e913be4b-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:06.650428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.650405    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465ea315-d67b-499d-828b-d020e913be4b-maas-api-xks-ca" pod "465ea315-d67b-499d-828b-d020e913be4b" (UID: "465ea315-d67b-499d-828b-d020e913be4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:32:06.650747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.650732    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465ea315-d67b-499d-828b-d020e913be4b-kube-api-access-8rpsk" pod "465ea315-d67b-499d-828b-d020e913be4b" (UID: "465ea315-d67b-499d-828b-d020e913be4b"). InnerVolumeSpecName "kube-api-access-8rpsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:06.750438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.750413    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ea315-d67b-499d-828b-d020e913be4b-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:06.750438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.750438    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rpsk\" (UniqueName: \"kubernetes.io/projected/465ea315-d67b-499d-828b-d020e913be4b-kube-api-access-8rpsk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:06.915926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:06.915287    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:07.414174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:07.414143    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"
Nov 03 13:32:07.414174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:07.414166    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8" event={"ID":"465ea315-d67b-499d-828b-d020e913be4b","Type":"ContainerDied","Data":"09c68ee044bbeca5e49416123df29fdfc44334e840c317e44fc6a8707dc076fb"}
Nov 03 13:32:07.414421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:07.414225    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09c68ee044bbeca5e49416123df29fdfc44334e840c317e44fc6a8707dc076fb"
Nov 03 13:32:07.547478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:07.547439    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:08.130201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:08.130121    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:08.731243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:08.729856    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:09.337735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:09.337701    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:09.940678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:09.940640    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:10.708948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:10.708900    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:10.789555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:10.789527    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:32:11.038565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:11.038489    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"]
Nov 03 13:32:11.047055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:11.047026    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935084bc-x4sb9"]
Nov 03 13:32:11.086508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:11.086473    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ee7256-cdfa-4c2f-99bc-e1f6c2602910" path="/var/lib/kubelet/pods/45ee7256-cdfa-4c2f-99bc-e1f6c2602910/volumes"
Nov 03 13:32:11.761205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:11.761170    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:12.758236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:12.755673    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:13.973231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:13.966265    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"]
Nov 03 13:32:13.973231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:13.967512    5420 cpu_manager.go:414] "containerMap: removing container" podUID="465ea315-d67b-499d-828b-d020e913be4b" containerName="revoke-keys"
Nov 03 13:32:13.973231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:13.967534    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465ea315-d67b-499d-828b-d020e913be4b" containerName="revoke-keys"
Nov 03 13:32:13.973231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:13.967712    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="465ea315-d67b-499d-828b-d020e913be4b" containerName="revoke-keys"
Nov 03 13:32:13.985238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:13.974822    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:13.985238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:13.982484    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:13.997233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:13.991276    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:32:13.997233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:13.991396    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:32:14.014241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.003442    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:32:14.014241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.003555    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:32:14.170247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.138277    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"]
Nov 03 13:32:14.170247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.166496    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82954635-4c89b\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:14.170247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.166794    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82954635-4c89b\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:14.170247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.166981    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf58s\" (UniqueName: \"kubernetes.io/projected/f129cc1f-30b8-4a3b-af67-1db7069e66d1-kube-api-access-qf58s\") pod \"maas-api-revoke-sub-82954635-4c89b\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:14.303603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.303509    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf58s\" (UniqueName: \"kubernetes.io/projected/f129cc1f-30b8-4a3b-af67-1db7069e66d1-kube-api-access-qf58s\") pod \"maas-api-revoke-sub-82954635-4c89b\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:14.304016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.303989    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82954635-4c89b\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:14.304302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.304271    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82954635-4c89b\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:14.305907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.305886    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82954635-4c89b\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:14.313260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.313238    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82954635-4c89b\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:14.318017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.317988    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf58s\" (UniqueName: \"kubernetes.io/projected/f129cc1f-30b8-4a3b-af67-1db7069e66d1-kube-api-access-qf58s\") pod \"maas-api-revoke-sub-82954635-4c89b\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:14.405539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.405494    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:14.411117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:14.411083    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:15.101254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:15.096491    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:15.350091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:15.350061    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"]
Nov 03 13:32:15.350777 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:32:15.350749    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf129cc1f_30b8_4a3b_af67_1db7069e66d1.slice/crio-7d0cc36b15bc2d918a3b28ac7bc620cf119b7ce318291080ad1ce356cdb1df21 WatchSource:0}: Error finding container 7d0cc36b15bc2d918a3b28ac7bc620cf119b7ce318291080ad1ce356cdb1df21: Status 404 returned error can't find the container with id 7d0cc36b15bc2d918a3b28ac7bc620cf119b7ce318291080ad1ce356cdb1df21
Nov 03 13:32:15.398170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:15.398150    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:15.619504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:15.619421    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b" event={"ID":"f129cc1f-30b8-4a3b-af67-1db7069e66d1","Type":"ContainerStarted","Data":"7d0cc36b15bc2d918a3b28ac7bc620cf119b7ce318291080ad1ce356cdb1df21"}
Nov 03 13:32:15.798017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:15.797972    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:16.416108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:16.416074    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:16.640619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:16.640530    5420 generic.go:356] "Generic (PLEG): container finished" podID="f129cc1f-30b8-4a3b-af67-1db7069e66d1" containerID="85f30a2265c23e9342d667a51955af6454c3dcacfe33b05835172a39bc42d5bb" exitCode=0
Nov 03 13:32:16.640974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:16.640894    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b" event={"ID":"f129cc1f-30b8-4a3b-af67-1db7069e66d1","Type":"ContainerDied","Data":"85f30a2265c23e9342d667a51955af6454c3dcacfe33b05835172a39bc42d5bb"}
Nov 03 13:32:17.220447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.220002    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:17.835042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.835015    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:17.863540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.863511    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:17.948254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.948156    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-openshift-ca\") pod \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") "
Nov 03 13:32:17.948467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.948449    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-xks-ca\") pod \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") "
Nov 03 13:32:17.948550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.948510    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f129cc1f-30b8-4a3b-af67-1db7069e66d1-kube-api-access-qf58s\" (UniqueName: \"kubernetes.io/projected/f129cc1f-30b8-4a3b-af67-1db7069e66d1-kube-api-access-qf58s\") pod \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\" (UID: \"f129cc1f-30b8-4a3b-af67-1db7069e66d1\") "
Nov 03 13:32:17.948550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.948507    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-openshift-ca" pod "f129cc1f-30b8-4a3b-af67-1db7069e66d1" (UID: "f129cc1f-30b8-4a3b-af67-1db7069e66d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:17.949486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.949457    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:17.950587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.950554    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-xks-ca" pod "f129cc1f-30b8-4a3b-af67-1db7069e66d1" (UID: "f129cc1f-30b8-4a3b-af67-1db7069e66d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:32:17.950718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:17.950698    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f129cc1f-30b8-4a3b-af67-1db7069e66d1-kube-api-access-qf58s" pod "f129cc1f-30b8-4a3b-af67-1db7069e66d1" (UID: "f129cc1f-30b8-4a3b-af67-1db7069e66d1"). InnerVolumeSpecName "kube-api-access-qf58s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:18.050839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:18.050812    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f129cc1f-30b8-4a3b-af67-1db7069e66d1-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:18.050839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:18.050839    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf58s\" (UniqueName: \"kubernetes.io/projected/f129cc1f-30b8-4a3b-af67-1db7069e66d1-kube-api-access-qf58s\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:18.238461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:18.238434    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:18.413819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:18.413268    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:18.661559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:18.661479    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"
Nov 03 13:32:18.661559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:18.661493    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b" event={"ID":"f129cc1f-30b8-4a3b-af67-1db7069e66d1","Type":"ContainerDied","Data":"7d0cc36b15bc2d918a3b28ac7bc620cf119b7ce318291080ad1ce356cdb1df21"}
Nov 03 13:32:18.661559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:18.661540    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d0cc36b15bc2d918a3b28ac7bc620cf119b7ce318291080ad1ce356cdb1df21"
Nov 03 13:32:19.016599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:19.016567    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:19.186871 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:19.186845    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:19.339066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:19.338982    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:19.800423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:19.800382    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:20.303917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:20.303887    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:20.454028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:20.453997    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:20.754470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:20.754442    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:20.913644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:20.913614    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:21.335168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:21.335135    5420 scope.go:122] "RemoveContainer" containerID="b40886127078d31ab74a5ade7f99b78c675e44f2be8a31777f27a0a4a83a3d1b"
Nov 03 13:32:21.346186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:21.346165    5420 scope.go:122] "RemoveContainer" containerID="8d6666b8baef02009d24d513153904fcc6c3233b7a8b561e7317603bb341d81a"
Nov 03 13:32:21.356534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:21.356517    5420 scope.go:122] "RemoveContainer" containerID="6c816b252f0f49141542af8de21eb0ab6b71eafdb3ed5bfce52122adb3d1d8e1"
Nov 03 13:32:21.536131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:21.536097    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:22.118499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.118467    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"]
Nov 03 13:32:22.119169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.119156    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f129cc1f-30b8-4a3b-af67-1db7069e66d1" containerName="revoke-keys"
Nov 03 13:32:22.119259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.119171    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f129cc1f-30b8-4a3b-af67-1db7069e66d1" containerName="revoke-keys"
Nov 03 13:32:22.119378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.119365    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f129cc1f-30b8-4a3b-af67-1db7069e66d1" containerName="revoke-keys"
Nov 03 13:32:22.125366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.125340    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.130013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.129985    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:32:22.131699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.131681    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:32:22.131793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.131736    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:32:22.131840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.131789    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:32:22.131956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.131942    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"]
Nov 03 13:32:22.223418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.223380    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe9b86c9-7vh99\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.223656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.223634    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe9b86c9-7vh99\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.223722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.223691    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmzpr\" (UniqueName: \"kubernetes.io/projected/157c9882-fa19-4a48-8e54-ad5ea40d0364-kube-api-access-kmzpr\") pod \"maas-api-revoke-sub-fe9b86c9-7vh99\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.324894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.324863    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmzpr\" (UniqueName: \"kubernetes.io/projected/157c9882-fa19-4a48-8e54-ad5ea40d0364-kube-api-access-kmzpr\") pod \"maas-api-revoke-sub-fe9b86c9-7vh99\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.325092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.325072    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe9b86c9-7vh99\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.325287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.325269    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe9b86c9-7vh99\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.325846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.325825    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe9b86c9-7vh99\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.327650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.327630    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe9b86c9-7vh99\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.333946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.333927    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmzpr\" (UniqueName: \"kubernetes.io/projected/157c9882-fa19-4a48-8e54-ad5ea40d0364-kube-api-access-kmzpr\") pod \"maas-api-revoke-sub-fe9b86c9-7vh99\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.425395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.425308    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:22.453275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.449057    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:22.780487 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:32:22.780454    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157c9882_fa19_4a48_8e54_ad5ea40d0364.slice/crio-14f840e06b0ee27441e10509d74a42625b0e5609233a08aa7278ce144c3e9449 WatchSource:0}: Error finding container 14f840e06b0ee27441e10509d74a42625b0e5609233a08aa7278ce144c3e9449: Status 404 returned error can't find the container with id 14f840e06b0ee27441e10509d74a42625b0e5609233a08aa7278ce144c3e9449
Nov 03 13:32:22.781457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:22.781426    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"]
Nov 03 13:32:23.226344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:23.226307    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:23.715021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:23.714978    5420 generic.go:356] "Generic (PLEG): container finished" podID="157c9882-fa19-4a48-8e54-ad5ea40d0364" containerID="8f87f0c103224985819662702bdf8075bb15063bddc03bff8540523d7c70cc44" exitCode=0
Nov 03 13:32:23.715423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:23.715180    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99" event={"ID":"157c9882-fa19-4a48-8e54-ad5ea40d0364","Type":"ContainerDied","Data":"8f87f0c103224985819662702bdf8075bb15063bddc03bff8540523d7c70cc44"}
Nov 03 13:32:23.715423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:23.715268    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99" event={"ID":"157c9882-fa19-4a48-8e54-ad5ea40d0364","Type":"ContainerStarted","Data":"14f840e06b0ee27441e10509d74a42625b0e5609233a08aa7278ce144c3e9449"}
Nov 03 13:32:23.844701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:23.844668    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:24.510509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:24.510161    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:24.733819 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:32:24.733783    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:32:24.892303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:24.892281    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:24.976816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:24.976785    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-xks-ca\") pod \"157c9882-fa19-4a48-8e54-ad5ea40d0364\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") "
Nov 03 13:32:24.976993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:24.976982    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157c9882-fa19-4a48-8e54-ad5ea40d0364-kube-api-access-kmzpr\" (UniqueName: \"kubernetes.io/projected/157c9882-fa19-4a48-8e54-ad5ea40d0364-kube-api-access-kmzpr\") pod \"157c9882-fa19-4a48-8e54-ad5ea40d0364\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") "
Nov 03 13:32:24.977073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:24.977057    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-openshift-ca\") pod \"157c9882-fa19-4a48-8e54-ad5ea40d0364\" (UID: \"157c9882-fa19-4a48-8e54-ad5ea40d0364\") "
Nov 03 13:32:24.977469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:24.977440    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-openshift-ca" pod "157c9882-fa19-4a48-8e54-ad5ea40d0364" (UID: "157c9882-fa19-4a48-8e54-ad5ea40d0364"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:24.977767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:24.977752    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:24.978886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:24.978856    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-xks-ca" pod "157c9882-fa19-4a48-8e54-ad5ea40d0364" (UID: "157c9882-fa19-4a48-8e54-ad5ea40d0364"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:32:24.979203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:24.979178    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157c9882-fa19-4a48-8e54-ad5ea40d0364-kube-api-access-kmzpr" pod "157c9882-fa19-4a48-8e54-ad5ea40d0364" (UID: "157c9882-fa19-4a48-8e54-ad5ea40d0364"). InnerVolumeSpecName "kube-api-access-kmzpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:25.079140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:25.079067    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/157c9882-fa19-4a48-8e54-ad5ea40d0364-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:25.079140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:25.079098    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmzpr\" (UniqueName: \"kubernetes.io/projected/157c9882-fa19-4a48-8e54-ad5ea40d0364-kube-api-access-kmzpr\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:25.114481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:25.114442    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:25.283958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:25.283928    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"]
Nov 03 13:32:25.380741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:25.380640    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:25.547235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:25.546687    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:25.792005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:25.785200    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99" event={"ID":"157c9882-fa19-4a48-8e54-ad5ea40d0364","Type":"ContainerDied","Data":"14f840e06b0ee27441e10509d74a42625b0e5609233a08aa7278ce144c3e9449"}
Nov 03 13:32:25.792005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:25.785279    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f840e06b0ee27441e10509d74a42625b0e5609233a08aa7278ce144c3e9449"
Nov 03 13:32:25.792005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:25.785353    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"
Nov 03 13:32:26.032486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:26.032449    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"]
Nov 03 13:32:26.036297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:26.036272    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:26.037702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:26.037680    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2709f34-5x4dv"]
Nov 03 13:32:26.181503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:26.181423    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:26.348234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:26.345350    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:27.017709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:27.017671    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:27.088397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:27.088366    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c" path="/var/lib/kubelet/pods/4cd5d8dd-8f5f-4a96-acc0-75b62ff1766c/volumes"
Nov 03 13:32:27.661394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:27.661356    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:28.355751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:28.355720    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:29.041524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:29.041458    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:29.681522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:29.681466    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:30.025919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.025886    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"]
Nov 03 13:32:30.030460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.030435    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5240bef2-75zgj"]
Nov 03 13:32:30.288960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.288841    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5" podUID="04f900ce-bd07-4fc3-9ab8-456c069204b8" containerName="payload-processing" containerID="cri-o://9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8" gracePeriod=25
Nov 03 13:32:30.300045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.299979    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:30.583135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.583110    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:30.695601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.695573    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-plugins-config-volume\") pod \"04f900ce-bd07-4fc3-9ab8-456c069204b8\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") "
Nov 03 13:32:30.696109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.695798    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-kube-api-access-2gpn2\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-kube-api-access-2gpn2\") pod \"04f900ce-bd07-4fc3-9ab8-456c069204b8\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") "
Nov 03 13:32:30.696109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.695863    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-provider-credentials\") pod \"04f900ce-bd07-4fc3-9ab8-456c069204b8\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") "
Nov 03 13:32:30.696109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.695892    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-routing-overlay-volume\") pod \"04f900ce-bd07-4fc3-9ab8-456c069204b8\" (UID: \"04f900ce-bd07-4fc3-9ab8-456c069204b8\") "
Nov 03 13:32:30.696109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.696046    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-plugins-config-volume" pod "04f900ce-bd07-4fc3-9ab8-456c069204b8" (UID: "04f900ce-bd07-4fc3-9ab8-456c069204b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:30.696326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.696303    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-routing-overlay-volume" pod "04f900ce-bd07-4fc3-9ab8-456c069204b8" (UID: "04f900ce-bd07-4fc3-9ab8-456c069204b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:30.696522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.696505    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:30.696562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.696525    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f900ce-bd07-4fc3-9ab8-456c069204b8-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:30.697821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.697801    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-kube-api-access-2gpn2" pod "04f900ce-bd07-4fc3-9ab8-456c069204b8" (UID: "04f900ce-bd07-4fc3-9ab8-456c069204b8"). InnerVolumeSpecName "kube-api-access-2gpn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:30.697898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.697884    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-provider-credentials" pod "04f900ce-bd07-4fc3-9ab8-456c069204b8" (UID: "04f900ce-bd07-4fc3-9ab8-456c069204b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:30.798222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.798174    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gpn2\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-kube-api-access-2gpn2\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:30.798222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.798224    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f900ce-bd07-4fc3-9ab8-456c069204b8-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:30.841537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.841444    5420 generic.go:356] "Generic (PLEG): container finished" podID="04f900ce-bd07-4fc3-9ab8-456c069204b8" containerID="9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8" exitCode=0
Nov 03 13:32:30.841537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.841493    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"
Nov 03 13:32:30.841537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.841521    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5" event={"ID":"04f900ce-bd07-4fc3-9ab8-456c069204b8","Type":"ContainerDied","Data":"9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8"}
Nov 03 13:32:30.841820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.841589    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5" event={"ID":"04f900ce-bd07-4fc3-9ab8-456c069204b8","Type":"ContainerDied","Data":"9a48209c129872a2d5a55519bf6d59525ffdbba224f9df2a5cd362f265e30cef"}
Nov 03 13:32:30.841820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.841619    5420 scope.go:122] "RemoveContainer" containerID="9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8"
Nov 03 13:32:30.854015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.853988    5420 scope.go:122] "RemoveContainer" containerID="9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8"
Nov 03 13:32:30.854323 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:32:30.854304    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8\": container with ID starting with 9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8 not found: ID does not exist" containerID="9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8"
Nov 03 13:32:30.854393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.854332    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8"} err="failed to get container status \"9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8\": rpc error: code = NotFound desc = could not find container \"9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8\": container with ID starting with 9459864220fb8126c1c234c9ee6b4e618fb3fe3726d8c61471d99c0da054dee8 not found: ID does not exist"
Nov 03 13:32:30.889403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.889376    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"]
Nov 03 13:32:30.893969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:30.893945    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-76b76swj5"]
Nov 03 13:32:31.040618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:31.040581    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:31.086457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:31.086429    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f900ce-bd07-4fc3-9ab8-456c069204b8" path="/var/lib/kubelet/pods/04f900ce-bd07-4fc3-9ab8-456c069204b8/volumes"
Nov 03 13:32:31.086936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:31.086922    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2df923-20e0-4c9c-bc84-94477db94d59" path="/var/lib/kubelet/pods/7a2df923-20e0-4c9c-bc84-94477db94d59/volumes"
Nov 03 13:32:31.495447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:31.495413    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:31.709981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:31.709948    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:32.044012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:32.043981    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:32.236004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:32.235972    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:32.951585 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:32.951536    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:33.600463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:33.600431    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:33.758680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:33.758644    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:34.078472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:34.078443    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:34.516232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:34.513796    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:34.679625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:34.679558    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:34.856518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:34.856416    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:35.448821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:35.448792    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:36.200309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:36.200273    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:36.956313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:36.956282    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:37.740933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:37.740844    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:38.440235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:38.433334    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:38.680132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:38.680087    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:39.661237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:39.660685    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:39.900236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:39.899749    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:40.642233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:40.642178    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:41.320854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:41.320825    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:41.662336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:41.662251    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:42.544847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:42.544815    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:43.023811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:43.023775    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"]
Nov 03 13:32:43.028282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:43.028256    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bb5bbd-vqwhl"]
Nov 03 13:32:43.087385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:43.087347    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a899091-6229-47e5-b778-7a668b28e4d9" path="/var/lib/kubelet/pods/3a899091-6229-47e5-b778-7a668b28e4d9/volumes"
Nov 03 13:32:43.195674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:43.195634    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:43.755159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:43.755132    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:44.433414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:44.433380    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:44.626572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:44.626521    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"]
Nov 03 13:32:44.958451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:44.958418    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:45.153718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:45.153686    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:45.420588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:45.420498    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:45.570380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:45.570336    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:45.998640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:45.998599    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:46.587668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:46.587635    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:47.300759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:47.300717    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:47.499238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:47.488289    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:48.178586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.178554    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"]
Nov 03 13:32:48.179329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.179313    5420 cpu_manager.go:414] "containerMap: removing container" podUID="04f900ce-bd07-4fc3-9ab8-456c069204b8" containerName="payload-processing"
Nov 03 13:32:48.179384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.179330    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f900ce-bd07-4fc3-9ab8-456c069204b8" containerName="payload-processing"
Nov 03 13:32:48.179384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.179340    5420 cpu_manager.go:414] "containerMap: removing container" podUID="157c9882-fa19-4a48-8e54-ad5ea40d0364" containerName="revoke-keys"
Nov 03 13:32:48.179384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.179345    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157c9882-fa19-4a48-8e54-ad5ea40d0364" containerName="revoke-keys"
Nov 03 13:32:48.179497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.179488    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f900ce-bd07-4fc3-9ab8-456c069204b8" containerName="payload-processing"
Nov 03 13:32:48.179533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.179499    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="157c9882-fa19-4a48-8e54-ad5ea40d0364" containerName="revoke-keys"
Nov 03 13:32:48.184473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.184455    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.192949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.192920    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"]
Nov 03 13:32:48.193509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.193165    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-c03d3bf9f10f\""
Nov 03 13:32:48.193509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.193273    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-ec4cc2d7b734\""
Nov 03 13:32:48.193509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.193312    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"e2e-cfl-e87c5e-provider-cred\""
Nov 03 13:32:48.194164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.194145    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"payload-processing-external-model-e2e--f52a173c-dockercfg-d5wkp\""
Nov 03 13:32:48.224682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.224650    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"kube-root-ca.crt\""
Nov 03 13:32:48.224682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.224653    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a62d22\""
Nov 03 13:32:48.224897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.224688    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"openshift-service-ca.crt\""
Nov 03 13:32:48.224897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.224690    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"routing-overlay\""
Nov 03 13:32:48.229388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.229368    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:48.237456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.237436    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.237557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.237485    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.237557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.237551    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdh86\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-kube-api-access-gdh86\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.237659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.237573    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.338651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.338614    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.339126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.338687    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.339126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.338732    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdh86\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-kube-api-access-gdh86\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.339126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.338767    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.339637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.339603    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.339729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.339635    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.341279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.341251    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.347155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.347130    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdh86\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-kube-api-access-gdh86\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.363291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.363257    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:48.501903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.501635    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:48.828802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.828719    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"]
Nov 03 13:32:48.863659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:48.863632    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"]
Nov 03 13:32:48.864770 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:32:48.864736    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f516b8_afbf_4543_a05f_4a0faa062ba9.slice/crio-ffa90ff00d31fe9324d68b6916f2f29dade8886c09982edd9073750ea1161756 WatchSource:0}: Error finding container ffa90ff00d31fe9324d68b6916f2f29dade8886c09982edd9073750ea1161756: Status 404 returned error can't find the container with id ffa90ff00d31fe9324d68b6916f2f29dade8886c09982edd9073750ea1161756
Nov 03 13:32:49.113285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.113155    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp" event={"ID":"d4f516b8-afbf-4543-a05f-4a0faa062ba9","Type":"ContainerStarted","Data":"1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a"}
Nov 03 13:32:49.113285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.113239    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp" event={"ID":"d4f516b8-afbf-4543-a05f-4a0faa062ba9","Type":"ContainerStarted","Data":"ffa90ff00d31fe9324d68b6916f2f29dade8886c09982edd9073750ea1161756"}
Nov 03 13:32:49.113285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.113290    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:49.142445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.142389    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp" podStartSLOduration=1.142368926 podStartE2EDuration="1.142368926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:49.140563843 +0000 UTC m=+1410.218328869" watchObservedRunningTime="2026-11-03 13:32:49.142368926 +0000 UTC m=+1410.220133952"
Nov 03 13:32:49.156075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.156040    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:49.373654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.373572    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:49.530418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.530380    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:49.627751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.627643    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks" podUID="7dc71620-7536-4e26-ac8e-f0f28e2e21b3" containerName="payload-processing" containerID="cri-o://76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1" gracePeriod=25
Nov 03 13:32:49.919666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.919643    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:49.996704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.996668    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-kube-api-access-zn7r4\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-kube-api-access-zn7r4\") pod \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") "
Nov 03 13:32:49.996916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.996785    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-plugins-config-volume\") pod \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") "
Nov 03 13:32:49.996994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.996927    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-provider-credentials\") pod \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") "
Nov 03 13:32:49.997164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.997131    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-routing-overlay-volume\") pod \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\" (UID: \"7dc71620-7536-4e26-ac8e-f0f28e2e21b3\") "
Nov 03 13:32:49.997487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.997289    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-plugins-config-volume" pod "7dc71620-7536-4e26-ac8e-f0f28e2e21b3" (UID: "7dc71620-7536-4e26-ac8e-f0f28e2e21b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:49.997998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.997721    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-routing-overlay-volume" pod "7dc71620-7536-4e26-ac8e-f0f28e2e21b3" (UID: "7dc71620-7536-4e26-ac8e-f0f28e2e21b3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:49.997998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.997950    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:49.997998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.997973    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:49.999557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.999524    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-kube-api-access-zn7r4" pod "7dc71620-7536-4e26-ac8e-f0f28e2e21b3" (UID: "7dc71620-7536-4e26-ac8e-f0f28e2e21b3"). InnerVolumeSpecName "kube-api-access-zn7r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:49.999645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:49.999552    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-provider-credentials" pod "7dc71620-7536-4e26-ac8e-f0f28e2e21b3" (UID: "7dc71620-7536-4e26-ac8e-f0f28e2e21b3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:50.101682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.101640    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn7r4\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-kube-api-access-zn7r4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:50.101682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.101682    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc71620-7536-4e26-ac8e-f0f28e2e21b3-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:50.124778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.124725    5420 generic.go:356] "Generic (PLEG): container finished" podID="7dc71620-7536-4e26-ac8e-f0f28e2e21b3" containerID="76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1" exitCode=0
Nov 03 13:32:50.124983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.124804    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"
Nov 03 13:32:50.125339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.124796    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks" event={"ID":"7dc71620-7536-4e26-ac8e-f0f28e2e21b3","Type":"ContainerDied","Data":"76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1"}
Nov 03 13:32:50.125702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.125676    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d95d887d-zpbks" event={"ID":"7dc71620-7536-4e26-ac8e-f0f28e2e21b3","Type":"ContainerDied","Data":"aca2a203d7e5578178a40009ad87ed7fdcd7b215a72c25d81bdc364a6864d0f4"}
Nov 03 13:32:50.125813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.125715    5420 scope.go:122] "RemoveContainer" containerID="76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1"
Nov 03 13:32:50.139746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.139721    5420 scope.go:122] "RemoveContainer" containerID="76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1"
Nov 03 13:32:50.140136 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:32:50.140104    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1\": container with ID starting with 76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1 not found: ID does not exist" containerID="76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1"
Nov 03 13:32:50.140244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.140165    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1"} err="failed to get container status \"76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1\": rpc error: code = NotFound desc = could not find container \"76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1\": container with ID starting with 76fb4f4c99b4c3e8e90d5a2a83dcf7519d52fc761c57cd266e6c3764167c38d1 not found: ID does not exist"
Nov 03 13:32:50.157089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.157062    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"]
Nov 03 13:32:50.161830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.161809    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-zpbks"]
Nov 03 13:32:50.290977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.290941    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:50.641379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.641348    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:50.680963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.680932    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"]
Nov 03 13:32:50.681679 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.681663    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7dc71620-7536-4e26-ac8e-f0f28e2e21b3" containerName="payload-processing"
Nov 03 13:32:50.681730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.681680    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc71620-7536-4e26-ac8e-f0f28e2e21b3" containerName="payload-processing"
Nov 03 13:32:50.681811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.681803    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc71620-7536-4e26-ac8e-f0f28e2e21b3" containerName="payload-processing"
Nov 03 13:32:50.687053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.687033    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:32:50.691883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.691859    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1ec4b892-dockercfg-l4whv\""
Nov 03 13:32:50.693283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.692485    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1ec4b892\""
Nov 03 13:32:50.696894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.696874    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"]
Nov 03 13:32:50.726269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.726237    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"]
Nov 03 13:32:50.731608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.731583    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:32:50.749257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.738104    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"]
Nov 03 13:32:50.823870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.823834    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:50.831127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.830524    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b079a4df-8022-4fdb-b26d-34791a3523cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc\" (UID: \"b079a4df-8022-4fdb-b26d-34791a3523cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:32:50.831127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.830635    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d5ed30-57ee-4d54-8402-7677917247d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm\" (UID: \"c7d5ed30-57ee-4d54-8402-7677917247d0\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:32:50.831127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.830869    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75fvs\" (UniqueName: \"kubernetes.io/projected/c7d5ed30-57ee-4d54-8402-7677917247d0-kube-api-access-75fvs\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm\" (UID: \"c7d5ed30-57ee-4d54-8402-7677917247d0\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:32:50.831127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.831005    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rng\" (UniqueName: \"kubernetes.io/projected/b079a4df-8022-4fdb-b26d-34791a3523cb-kube-api-access-p2rng\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc\" (UID: \"b079a4df-8022-4fdb-b26d-34791a3523cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:32:50.932059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.931980    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d5ed30-57ee-4d54-8402-7677917247d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm\" (UID: \"c7d5ed30-57ee-4d54-8402-7677917247d0\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:32:50.932264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.932131    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75fvs\" (UniqueName: \"kubernetes.io/projected/c7d5ed30-57ee-4d54-8402-7677917247d0-kube-api-access-75fvs\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm\" (UID: \"c7d5ed30-57ee-4d54-8402-7677917247d0\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:32:50.932347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.932303    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rng\" (UniqueName: \"kubernetes.io/projected/b079a4df-8022-4fdb-b26d-34791a3523cb-kube-api-access-p2rng\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc\" (UID: \"b079a4df-8022-4fdb-b26d-34791a3523cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:32:50.932406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.932380    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b079a4df-8022-4fdb-b26d-34791a3523cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc\" (UID: \"b079a4df-8022-4fdb-b26d-34791a3523cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:32:50.933010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.932983    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d5ed30-57ee-4d54-8402-7677917247d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm\" (UID: \"c7d5ed30-57ee-4d54-8402-7677917247d0\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:32:50.933363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.933340    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b079a4df-8022-4fdb-b26d-34791a3523cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc\" (UID: \"b079a4df-8022-4fdb-b26d-34791a3523cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:32:50.942256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.942185    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75fvs\" (UniqueName: \"kubernetes.io/projected/c7d5ed30-57ee-4d54-8402-7677917247d0-kube-api-access-75fvs\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm\" (UID: \"c7d5ed30-57ee-4d54-8402-7677917247d0\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:32:50.942685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:50.942664    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rng\" (UniqueName: \"kubernetes.io/projected/b079a4df-8022-4fdb-b26d-34791a3523cb-kube-api-access-p2rng\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc\" (UID: \"b079a4df-8022-4fdb-b26d-34791a3523cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:32:51.005283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:51.005246    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:32:51.083166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:51.081504    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:32:51.168593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:51.163293    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc71620-7536-4e26-ac8e-f0f28e2e21b3" path="/var/lib/kubelet/pods/7dc71620-7536-4e26-ac8e-f0f28e2e21b3/volumes"
Nov 03 13:32:51.202027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:51.201942    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:51.308435 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:32:51.308398    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb079a4df_8022_4fdb_b26d_34791a3523cb.slice/crio-524c930a251927f3ec03010f53a0315dbf99632b945e7f32323091cccfbe9b2e WatchSource:0}: Error finding container 524c930a251927f3ec03010f53a0315dbf99632b945e7f32323091cccfbe9b2e: Status 404 returned error can't find the container with id 524c930a251927f3ec03010f53a0315dbf99632b945e7f32323091cccfbe9b2e
Nov 03 13:32:51.349343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:51.349154    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"]
Nov 03 13:32:51.369098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:51.369024    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:51.453781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:51.453750    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"]
Nov 03 13:32:51.455396 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:32:51.455341    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d5ed30_57ee_4d54_8402_7677917247d0.slice/crio-058f17bff346c6346acd46c8219ca78c996438b9593ac0cae037e48dfbc5654d WatchSource:0}: Error finding container 058f17bff346c6346acd46c8219ca78c996438b9593ac0cae037e48dfbc5654d: Status 404 returned error can't find the container with id 058f17bff346c6346acd46c8219ca78c996438b9593ac0cae037e48dfbc5654d
Nov 03 13:32:52.130726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.129390    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:52.182838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.182807    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm" event={"ID":"c7d5ed30-57ee-4d54-8402-7677917247d0","Type":"ContainerStarted","Data":"3d07d7ab64981acd07a68b3dc02b78f405506cf14d65b84de157509719d9df4d"}
Nov 03 13:32:52.183000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.182861    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm" event={"ID":"c7d5ed30-57ee-4d54-8402-7677917247d0","Type":"ContainerStarted","Data":"058f17bff346c6346acd46c8219ca78c996438b9593ac0cae037e48dfbc5654d"}
Nov 03 13:32:52.183000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.182885    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:32:52.184580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.184556    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc" event={"ID":"b079a4df-8022-4fdb-b26d-34791a3523cb","Type":"ContainerStarted","Data":"0041b2dde555d9068b77a7573369ee6f03a1279fece86da7ccf9964e547d2b46"}
Nov 03 13:32:52.184695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.184595    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc" event={"ID":"b079a4df-8022-4fdb-b26d-34791a3523cb","Type":"ContainerStarted","Data":"524c930a251927f3ec03010f53a0315dbf99632b945e7f32323091cccfbe9b2e"}
Nov 03 13:32:52.184695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.184615    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:32:52.203638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.203582    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm" podStartSLOduration=2.2035657300000002 podStartE2EDuration="2.20356573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:52.201414399 +0000 UTC m=+1413.279179428" watchObservedRunningTime="2026-11-03 13:32:52.20356573 +0000 UTC m=+1413.281330757"
Nov 03 13:32:52.224303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.224183    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc" podStartSLOduration=2.224160554 podStartE2EDuration="2.224160554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:52.219539203 +0000 UTC m=+1413.297304230" watchObservedRunningTime="2026-11-03 13:32:52.224160554 +0000 UTC m=+1413.301925582"
Nov 03 13:32:52.891923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:52.891894    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:53.036651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:53.036616    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"]
Nov 03 13:32:53.041243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:53.041193    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ce8aa5-6fjkx"]
Nov 03 13:32:53.086111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:53.086077    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23cb02a4-b31f-45c6-8763-5b479b788a3f" path="/var/lib/kubelet/pods/23cb02a4-b31f-45c6-8763-5b479b788a3f/volumes"
Nov 03 13:32:53.430101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:53.429280    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:54.091801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.091023    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:54.115243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.114277    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp" podUID="d4f516b8-afbf-4543-a05f-4a0faa062ba9" containerName="payload-processing" containerID="cri-o://1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a" gracePeriod=25
Nov 03 13:32:54.416721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.416695    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:54.514904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.514875    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-kube-api-access-gdh86\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-kube-api-access-gdh86\") pod \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") "
Nov 03 13:32:54.515320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.514916    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-provider-credentials\") pod \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") "
Nov 03 13:32:54.515320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.515069    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-plugins-config-volume\") pod \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") "
Nov 03 13:32:54.515320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.515294    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-routing-overlay-volume\") pod \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\" (UID: \"d4f516b8-afbf-4543-a05f-4a0faa062ba9\") "
Nov 03 13:32:54.515610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.515576    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-plugins-config-volume" pod "d4f516b8-afbf-4543-a05f-4a0faa062ba9" (UID: "d4f516b8-afbf-4543-a05f-4a0faa062ba9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:54.515696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.515641    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-routing-overlay-volume" pod "d4f516b8-afbf-4543-a05f-4a0faa062ba9" (UID: "d4f516b8-afbf-4543-a05f-4a0faa062ba9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:32:54.515960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.515943    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:54.516001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.515963    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4f516b8-afbf-4543-a05f-4a0faa062ba9-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:54.517724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.517694    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-provider-credentials" pod "d4f516b8-afbf-4543-a05f-4a0faa062ba9" (UID: "d4f516b8-afbf-4543-a05f-4a0faa062ba9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:54.517808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.517735    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-kube-api-access-gdh86" pod "d4f516b8-afbf-4543-a05f-4a0faa062ba9" (UID: "d4f516b8-afbf-4543-a05f-4a0faa062ba9"). InnerVolumeSpecName "kube-api-access-gdh86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:32:54.617217 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.617172    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdh86\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-kube-api-access-gdh86\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:54.617397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.617206    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4f516b8-afbf-4543-a05f-4a0faa062ba9-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:32:54.734691 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:32:54.734651    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:32:54.792916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:54.792885    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:55.211437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.211401    5420 generic.go:356] "Generic (PLEG): container finished" podID="d4f516b8-afbf-4543-a05f-4a0faa062ba9" containerID="1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a" exitCode=0
Nov 03 13:32:55.211606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.211447    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"
Nov 03 13:32:55.211606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.211485    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp" event={"ID":"d4f516b8-afbf-4543-a05f-4a0faa062ba9","Type":"ContainerDied","Data":"1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a"}
Nov 03 13:32:55.211606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.211544    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp" event={"ID":"d4f516b8-afbf-4543-a05f-4a0faa062ba9","Type":"ContainerDied","Data":"ffa90ff00d31fe9324d68b6916f2f29dade8886c09982edd9073750ea1161756"}
Nov 03 13:32:55.211606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.211571    5420 scope.go:122] "RemoveContainer" containerID="1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a"
Nov 03 13:32:55.223041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.223020    5420 scope.go:122] "RemoveContainer" containerID="1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a"
Nov 03 13:32:55.223342 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:32:55.223314    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a\": container with ID starting with 1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a not found: ID does not exist" containerID="1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a"
Nov 03 13:32:55.223395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.223342    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a"} err="failed to get container status \"1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a\": rpc error: code = NotFound desc = could not find container \"1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a\": container with ID starting with 1cd30b419dc49ba4fc32eefb6176b221e96482a8e1d192c7ca4648029ca3ee8a not found: ID does not exist"
Nov 03 13:32:55.232485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.232463    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"]
Nov 03 13:32:55.238658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.238635    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4mwmfp"]
Nov 03 13:32:55.525240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:55.525055    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:56.271010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:56.269620    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:56.952666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:56.952634    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:57.088690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:57.088659    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f516b8-afbf-4543-a05f-4a0faa062ba9" path="/var/lib/kubelet/pods/d4f516b8-afbf-4543-a05f-4a0faa062ba9/volumes"
Nov 03 13:32:57.816812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:57.816775    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:58.402242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:58.399857    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:59.000415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:58.996858    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:59.722832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:32:59.719990    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:00.315517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:00.315481    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:00.481188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:00.481156    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:01.088636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:01.088604    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:33:01.177268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:01.177195    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:02.071313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:02.071269    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:02.710235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:02.707708    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:03.196203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:03.196122    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:33:03.197157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:03.197135    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:33:03.418115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:03.413665    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"]
Nov 03 13:33:03.438237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:03.435881    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy" containerID="cri-o://82f58f5e7a3b97a73fe6199a2380d42d7b5a7c8d8764fc45a57728ad8098f761" gracePeriod=30
Nov 03 13:33:03.632581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:03.630747    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:04.618301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:04.618264    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:04.990239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:04.986889    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:05.589875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:05.589840    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:05.818536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:05.818498    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:05.891491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:05.891367    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:33:05.891822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:05.891790    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:33:06.009861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.009825    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"]
Nov 03 13:33:06.012241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.012193    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d4f516b8-afbf-4543-a05f-4a0faa062ba9" containerName="payload-processing"
Nov 03 13:33:06.013302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.013275    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f516b8-afbf-4543-a05f-4a0faa062ba9" containerName="payload-processing"
Nov 03 13:33:06.013549 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.013531    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f516b8-afbf-4543-a05f-4a0faa062ba9" containerName="payload-processing"
Nov 03 13:33:06.041236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.035422    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"]
Nov 03 13:33:06.041236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.035556    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.060574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.060541    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-418af8-openshift-default-dockercfg-ksq77\""
Nov 03 13:33:06.164320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.156171    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:06.233398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.199383    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-podinfo\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.233847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.233819    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmlq\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-kube-api-access-nqmlq\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.234021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.234003    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-token\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.261542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.261501    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-envoy\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.261844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.261791    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-credential-socket\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.262029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.262009    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-certs\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.262191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.262173    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-data\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.262552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.262531    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istiod-ca-cert\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.262756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.262736    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-socket\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.371582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.371543    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-credential-socket\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.371869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.371846    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-certs\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.372022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.371999    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-data\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.372261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.372246    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istiod-ca-cert\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.372438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.372422    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-socket\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.372778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.372753    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-podinfo\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.373001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.372982    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmlq\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-kube-api-access-nqmlq\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.373136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.373120    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-token\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.373290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.373273    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-envoy\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.386235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.376186    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istiod-ca-cert\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.403685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.388181    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-credential-socket\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.408246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.390568    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-certs\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.408454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.390635    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-data\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.408567 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.390816    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-socket\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.411197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.411163    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-envoy\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.427286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.423841    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-podinfo\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.433620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.428156    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-token\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.442581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.442549    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmlq\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-kube-api-access-nqmlq\") pod \"e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") " pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.733039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.733012    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:06.838268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:06.836628    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:07.035362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.035275    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"]
Nov 03 13:33:07.041698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.041664    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"]
Nov 03 13:33:07.043421 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:33:07.043389    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cc5a21_b6ec_49b4_8d77_51fa0a5a1710.slice/crio-cf12048ffffdae121709e25efd67d6359f07e1694bc207d66cb6769c7ddd517f WatchSource:0}: Error finding container cf12048ffffdae121709e25efd67d6359f07e1694bc207d66cb6769c7ddd517f: Status 404 returned error can't find the container with id cf12048ffffdae121709e25efd67d6359f07e1694bc207d66cb6769c7ddd517f
Nov 03 13:33:07.047374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.047301    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:07.047789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.047756    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:07.047888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.047825    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:07.048026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.047894    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:07.048099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.048027    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539e8ee4-7b84m"]
Nov 03 13:33:07.088416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.088385    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1885eb48-0882-4f66-ba99-27bae5c8a95f" path="/var/lib/kubelet/pods/1885eb48-0882-4f66-ba99-27bae5c8a95f/volumes"
Nov 03 13:33:07.695781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.695745    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" event={"ID":"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710","Type":"ContainerStarted","Data":"8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103"}
Nov 03 13:33:07.695974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.695830    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" event={"ID":"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710","Type":"ContainerStarted","Data":"cf12048ffffdae121709e25efd67d6359f07e1694bc207d66cb6769c7ddd517f"}
Nov 03 13:33:07.733838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.733779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:07.734040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.733910    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:07.739652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:07.739594    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" podStartSLOduration=2.739577204 podStartE2EDuration="2.739577204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:07.733024043 +0000 UTC m=+1428.810789072" watchObservedRunningTime="2026-11-03 13:33:07.739577204 +0000 UTC m=+1428.817342294"
Nov 03 13:33:08.351651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.351601    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:08.722185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.722152    5420 generic.go:356] "Generic (PLEG): container finished" podID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerID="82f58f5e7a3b97a73fe6199a2380d42d7b5a7c8d8764fc45a57728ad8098f761" exitCode=0
Nov 03 13:33:08.722412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.722245    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" event={"ID":"02a0ce67-5f7d-4bce-afe1-ab40efbf422c","Type":"ContainerDied","Data":"82f58f5e7a3b97a73fe6199a2380d42d7b5a7c8d8764fc45a57728ad8098f761"}
Nov 03 13:33:08.741521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.741485    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:08.793139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.793109    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:33:08.852365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.852330    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:08.948507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.948429    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-envoy\") pod \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") "
Nov 03 13:33:08.948507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.948466    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istiod-ca-cert\") pod \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") "
Nov 03 13:33:08.948507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.948508    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-socket\") pod \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") "
Nov 03 13:33:08.948760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.948637    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-token\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-token\") pod \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") "
Nov 03 13:33:08.948760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.948734    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-podinfo\") pod \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") "
Nov 03 13:33:08.948878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.948831    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-certs\") pod \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") "
Nov 03 13:33:08.948945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.948889    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-socket" pod "02a0ce67-5f7d-4bce-afe1-ab40efbf422c" (UID: "02a0ce67-5f7d-4bce-afe1-ab40efbf422c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:33:08.948945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.948933    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istiod-ca-cert" pod "02a0ce67-5f7d-4bce-afe1-ab40efbf422c" (UID: "02a0ce67-5f7d-4bce-afe1-ab40efbf422c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:33:08.949140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.949116    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-certs" pod "02a0ce67-5f7d-4bce-afe1-ab40efbf422c" (UID: "02a0ce67-5f7d-4bce-afe1-ab40efbf422c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:33:08.949225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.949172    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-credential-socket\") pod \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") "
Nov 03 13:33:08.949295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.949263    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-data\") pod \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") "
Nov 03 13:33:08.949556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.949371    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-kube-api-access-x6pnc\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-kube-api-access-x6pnc\") pod \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\" (UID: \"02a0ce67-5f7d-4bce-afe1-ab40efbf422c\") "
Nov 03 13:33:08.949556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.949381    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-credential-socket" pod "02a0ce67-5f7d-4bce-afe1-ab40efbf422c" (UID: "02a0ce67-5f7d-4bce-afe1-ab40efbf422c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:33:08.949556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.949531    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-data" pod "02a0ce67-5f7d-4bce-afe1-ab40efbf422c" (UID: "02a0ce67-5f7d-4bce-afe1-ab40efbf422c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:33:08.950244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.950185    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:08.950244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.950227    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:08.950244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.950242    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:08.950505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.950256    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:08.950505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.950271    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:08.950979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.950943    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-envoy" pod "02a0ce67-5f7d-4bce-afe1-ab40efbf422c" (UID: "02a0ce67-5f7d-4bce-afe1-ab40efbf422c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:33:08.951079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.951060    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-token" pod "02a0ce67-5f7d-4bce-afe1-ab40efbf422c" (UID: "02a0ce67-5f7d-4bce-afe1-ab40efbf422c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:33:08.951282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.951184    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-podinfo" pod "02a0ce67-5f7d-4bce-afe1-ab40efbf422c" (UID: "02a0ce67-5f7d-4bce-afe1-ab40efbf422c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:33:08.951685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:08.951661    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-kube-api-access-x6pnc" pod "02a0ce67-5f7d-4bce-afe1-ab40efbf422c" (UID: "02a0ce67-5f7d-4bce-afe1-ab40efbf422c"). InnerVolumeSpecName "kube-api-access-x6pnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:33:09.011874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.011846    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:09.052374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.052343    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6pnc\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-kube-api-access-x6pnc\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:09.052374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.052372    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:09.052576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.052383    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:09.052576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.052391    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02a0ce67-5f7d-4bce-afe1-ab40efbf422c-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:33:09.164102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.164072    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:09.732422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.732392    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"
Nov 03 13:33:09.732884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.732422    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn" event={"ID":"02a0ce67-5f7d-4bce-afe1-ab40efbf422c","Type":"ContainerDied","Data":"fdc2a0ba9046450db78cb12aa7bd4eefb5850a975a1eeea7eec7f7096e308a42"}
Nov 03 13:33:09.732884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.732481    5420 scope.go:122] "RemoveContainer" containerID="82f58f5e7a3b97a73fe6199a2380d42d7b5a7c8d8764fc45a57728ad8098f761"
Nov 03 13:33:09.733035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.733018    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:09.734244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.734189    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:09.758496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.758467    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"]
Nov 03 13:33:09.762919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.762893    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-775hn"]
Nov 03 13:33:09.954338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:09.954301    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:10.209820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:10.209793    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:10.356348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:10.356311    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:10.856035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:10.855994    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:11.086517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.086457    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" path="/var/lib/kubelet/pods/02a0ce67-5f7d-4bce-afe1-ab40efbf422c/volumes"
Nov 03 13:33:11.507861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.507825    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:11.536732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.536699    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"]
Nov 03 13:33:11.537445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.537430    5420 cpu_manager.go:414] "containerMap: removing container" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy"
Nov 03 13:33:11.537507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.537446    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy"
Nov 03 13:33:11.537574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.537564    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a0ce67-5f7d-4bce-afe1-ab40efbf422c" containerName="istio-proxy"
Nov 03 13:33:11.543783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.543754    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:11.551264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.551232    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"]
Nov 03 13:33:11.551571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.551541    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-418af8-dockercfg-9ppxq\""
Nov 03 13:33:11.551571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.551560    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-418af8\""
Nov 03 13:33:11.670576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.670542    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:11.696339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.696305    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxj2\" (UniqueName: \"kubernetes.io/projected/8d139438-743f-4c7c-a0f3-1ded420d676b-kube-api-access-hlxj2\") pod \"payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76\" (UID: \"8d139438-743f-4c7c-a0f3-1ded420d676b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:11.696554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.696539    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d139438-743f-4c7c-a0f3-1ded420d676b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76\" (UID: \"8d139438-743f-4c7c-a0f3-1ded420d676b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:11.801288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.800388    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d139438-743f-4c7c-a0f3-1ded420d676b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76\" (UID: \"8d139438-743f-4c7c-a0f3-1ded420d676b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:11.801288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.800552    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxj2\" (UniqueName: \"kubernetes.io/projected/8d139438-743f-4c7c-a0f3-1ded420d676b-kube-api-access-hlxj2\") pod \"payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76\" (UID: \"8d139438-743f-4c7c-a0f3-1ded420d676b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:11.805268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.802160    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d139438-743f-4c7c-a0f3-1ded420d676b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76\" (UID: \"8d139438-743f-4c7c-a0f3-1ded420d676b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:11.846233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.826107    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:11.859254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.855350    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxj2\" (UniqueName: \"kubernetes.io/projected/8d139438-743f-4c7c-a0f3-1ded420d676b-kube-api-access-hlxj2\") pod \"payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76\" (UID: \"8d139438-743f-4c7c-a0f3-1ded420d676b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:11.882233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:11.878505    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:12.255361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:12.255332    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"]
Nov 03 13:33:12.256197 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:33:12.256170    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d139438_743f_4c7c_a0f3_1ded420d676b.slice/crio-ad050b4bbb56232b9be3288869f53e9e797e5489cee57cf0eae1d7c46dddf5b6 WatchSource:0}: Error finding container ad050b4bbb56232b9be3288869f53e9e797e5489cee57cf0eae1d7c46dddf5b6: Status 404 returned error can't find the container with id ad050b4bbb56232b9be3288869f53e9e797e5489cee57cf0eae1d7c46dddf5b6
Nov 03 13:33:12.416299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:12.409985    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:12.802336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:12.802302    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76" event={"ID":"8d139438-743f-4c7c-a0f3-1ded420d676b","Type":"ContainerStarted","Data":"1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215"}
Nov 03 13:33:12.802535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:12.802352    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76" event={"ID":"8d139438-743f-4c7c-a0f3-1ded420d676b","Type":"ContainerStarted","Data":"ad050b4bbb56232b9be3288869f53e9e797e5489cee57cf0eae1d7c46dddf5b6"}
Nov 03 13:33:12.802535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:12.802446    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:12.822701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:12.822648    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76" podStartSLOduration=1.8226324919999999 podStartE2EDuration="1.822632492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:12.81861339 +0000 UTC m=+1433.896378418" watchObservedRunningTime="2026-11-03 13:33:12.822632492 +0000 UTC m=+1433.900397517"
Nov 03 13:33:13.263524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:13.263461    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:14.010360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:14.007332    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:14.887235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:14.886678    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:15.630809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:15.630772    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:16.240234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:16.224850    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:16.819784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:16.818835    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:17.446247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:17.446166    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:18.133053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:18.131779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:18.376375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:18.376345    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:18.718745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:18.718714    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:19.025526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:19.025446    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"]
Nov 03 13:33:19.029671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:19.029647    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa4505f-9jnln"]
Nov 03 13:33:19.086085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:19.086054    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2543bc1f-d0ea-4a00-9958-4b20958915b2" path="/var/lib/kubelet/pods/2543bc1f-d0ea-4a00-9958-4b20958915b2/volumes"
Nov 03 13:33:19.320013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:19.319910    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:19.774900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:19.774870    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:20.013128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:20.013090    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:20.870270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:20.870234    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:21.420449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:21.420416    5420 scope.go:122] "RemoveContainer" containerID="9ec4c50f99f9b7e5c4ce00f90741a2587411c92780fe732987aad22aed4656bf"
Nov 03 13:33:21.431728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:21.431705    5420 scope.go:122] "RemoveContainer" containerID="6c9df22a1d988f2c9f5b108f725e7b945c151a441417c02b207f4a576dfaa1fe"
Nov 03 13:33:21.445964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:21.443427    5420 scope.go:122] "RemoveContainer" containerID="14705518bd7fe68e582be63e8a3ada05b208f0bcb32372094feb84062788f880"
Nov 03 13:33:21.455283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:21.455264    5420 scope.go:122] "RemoveContainer" containerID="e0a842660f8d4794ae3cb57eeeaf4035006ebd0a3989ab10050b3930286428b5"
Nov 03 13:33:21.479308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:21.479275    5420 scope.go:122] "RemoveContainer" containerID="2598fca336410e7fc3e801ee6807c1ef936e8bb317296af49034599d7af65fa4"
Nov 03 13:33:22.224303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:22.224267    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:22.359652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:22.359622    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:22.691583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:22.691500    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:23.020320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:23.019790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:23.757240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:23.756711    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:23.835268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:23.835199    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:33:24.633349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:24.633320    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:24.736741 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:33:24.736706    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:33:25.570777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:25.566841    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:26.221497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:26.221422    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:26.971356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:26.971305    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:27.653878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:27.653836    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:28.297305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:28.293967    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:28.884625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:28.884589    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:29.500568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:29.500535    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:29.755324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:29.755244    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:30.232534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:30.232499    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:30.851234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:30.849408    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:31.024501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:31.024464    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"]
Nov 03 13:33:31.028980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:31.028947    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958a8579-g72n9"]
Nov 03 13:33:31.089062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:31.089028    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd045566-c1a6-468a-9a18-f8b92430f049" path="/var/lib/kubelet/pods/bd045566-c1a6-468a-9a18-f8b92430f049/volumes"
Nov 03 13:33:31.670010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:31.669976    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:32.324994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:32.324963    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:32.913267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:32.909620    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:33.040900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:33.040862    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:33:33.995236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:33.995117    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:34.954234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:34.945498    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:35.158810 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.158778    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"]
Nov 03 13:33:35.163348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.163329    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.172194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.172167    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"]
Nov 03 13:33:35.173832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.173812    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a62d22-openshift-default-dockercfg-r9srg\""
Nov 03 13:33:35.206941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.206867    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.206941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.206916    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-data\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.207111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.207014    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-certs\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.207111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.207081    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m5c4\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-kube-api-access-5m5c4\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.207111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.207105    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-token\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.207287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.207270    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-credential-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.207368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.207353    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-envoy\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.207408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.207379    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f6377b-b159-4b6c-85c6-ad75b000c640-istiod-ca-cert\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.207538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.207522    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-podinfo\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.308767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.308734    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-certs\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.308943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.308827    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m5c4\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-kube-api-access-5m5c4\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.308943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.308862    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-token\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.308943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.308929    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-credential-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309021    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-envoy\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309051    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f6377b-b159-4b6c-85c6-ad75b000c640-istiod-ca-cert\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309184    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-certs\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309206    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-podinfo\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309292    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309332    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-data\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309414    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-credential-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309634    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309659    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-data\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.309830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.309729    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f6377b-b159-4b6c-85c6-ad75b000c640-istiod-ca-cert\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.311343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.311321    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-envoy\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.311592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.311576    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-podinfo\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.352541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.352507    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-token\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.353019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.352972    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m5c4\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-kube-api-access-5m5c4\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.481886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.481518    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:35.493956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.493591    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:35.741334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.741141    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:35.855140 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:33:35.854584    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f6377b_b159_4b6c_85c6_ad75b000c640.slice/crio-fb53e608fda3ddab71b26ca291744cf01506e2928fe9e46f87f3d14fabd0bf6c WatchSource:0}: Error finding container fb53e608fda3ddab71b26ca291744cf01506e2928fe9e46f87f3d14fabd0bf6c: Status 404 returned error can't find the container with id fb53e608fda3ddab71b26ca291744cf01506e2928fe9e46f87f3d14fabd0bf6c
Nov 03 13:33:35.858061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.858033    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"]
Nov 03 13:33:35.859896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.859858    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:35.860010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.859936    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:35.860010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:35.859987    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:36.098028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:36.097924    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" event={"ID":"d1f6377b-b159-4b6c-85c6-ad75b000c640","Type":"ContainerStarted","Data":"4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc"}
Nov 03 13:33:36.098028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:36.097971    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" event={"ID":"d1f6377b-b159-4b6c-85c6-ad75b000c640","Type":"ContainerStarted","Data":"fb53e608fda3ddab71b26ca291744cf01506e2928fe9e46f87f3d14fabd0bf6c"}
Nov 03 13:33:36.130237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:36.125925    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" podStartSLOduration=1.125908125 podStartE2EDuration="1.125908125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:36.119724299 +0000 UTC m=+1457.197489333" watchObservedRunningTime="2026-11-03 13:33:36.125908125 +0000 UTC m=+1457.203673190"
Nov 03 13:33:36.483294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:36.483263    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:36.484654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:36.484632    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:33:36.484744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:36.484676    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 13:33:36.519834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:36.519797    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:37.108665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:37.108389    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:37.483734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:37.483695    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:33:37.484284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:37.483769    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 13:33:37.907779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:37.907700    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:38.482746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:38.482709    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:33:38.482941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:38.482768    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 13:33:38.530284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:38.525440    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:39.337231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:39.331744    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:39.484181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:39.483608    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:33:39.484181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:39.483672    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 13:33:39.812512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:39.812475    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:39.960238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:39.956774    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:40.483107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:40.483071    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:33:40.483318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:40.483124    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 13:33:40.606615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:40.606574    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:41.349691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:41.349658    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:41.482667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:41.482624    5420 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Nov 03 13:33:41.482860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:41.482706    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Nov 03 13:33:42.133937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:42.133851    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:42.558671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:42.558638    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:42.560120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:42.560093    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:42.566159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:42.566134    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:33:42.717493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:42.717465    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:43.382276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:43.380288    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:43.985602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:43.985569    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:44.748281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:44.741307    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:45.245188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.245151    5420 generic.go:356] "Generic (PLEG): container finished" podID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerID="8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103" exitCode=0
Nov 03 13:33:45.245626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.245245    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" event={"ID":"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710","Type":"ContainerDied","Data":"8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103"}
Nov 03 13:33:45.245681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.245666    5420 scope.go:122] "RemoveContainer" containerID="8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103"
Nov 03 13:33:45.247575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.247543    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:45.247690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.247612    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:45.247690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.247645    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:45.439285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.439203    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"]
Nov 03 13:33:45.444065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.444044    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.451553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.451517    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1ec4b892-openshift-default-dockercfg-rm5mf\""
Nov 03 13:33:45.455612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.455589    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"]
Nov 03 13:33:45.466880 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.466668    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf24j\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-kube-api-access-vf24j\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.466880 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.466725    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-token\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.467076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.466896    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-data\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.467076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.466988    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-credential-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.467223 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.467128    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.468458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.467281    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-certs\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.468458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.467537    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5edc82-7131-4b51-b078-ca321b6d1e16-istiod-ca-cert\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.468458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.467654    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-envoy\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.468458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.467859    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-podinfo\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576229    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-certs\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576421    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5edc82-7131-4b51-b078-ca321b6d1e16-istiod-ca-cert\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576490    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-envoy\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576573    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-certs\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576590    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-podinfo\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576648    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf24j\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-kube-api-access-vf24j\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576695    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-token\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576737    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-data\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576770    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-credential-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.576841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.576833    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.577128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.577108    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-data\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.577225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.577188    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-credential-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.577283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.577265    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.577520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.577496    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5edc82-7131-4b51-b078-ca321b6d1e16-istiod-ca-cert\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.578899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.578875    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-envoy\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.579807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.579786    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-podinfo\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.586063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.586033    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf24j\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-kube-api-access-vf24j\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.586150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.586042    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-token\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.763866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.763826    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:45.855233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:45.854817    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:46.288601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.288572    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:46.289725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.289699    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" event={"ID":"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710","Type":"ContainerStarted","Data":"036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0"}
Nov 03 13:33:46.372535 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:33:46.372508    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5edc82_7131_4b51_b078_ca321b6d1e16.slice/crio-f82f836af664650f385fa03f88a50ccd60346f9f19450064f657e4a1b15ce87e WatchSource:0}: Error finding container f82f836af664650f385fa03f88a50ccd60346f9f19450064f657e4a1b15ce87e: Status 404 returned error can't find the container with id f82f836af664650f385fa03f88a50ccd60346f9f19450064f657e4a1b15ce87e
Nov 03 13:33:46.372761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.372739    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"]
Nov 03 13:33:46.375574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.375537    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:46.375692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.375618    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:46.375692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.375668    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:33:46.453858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.453827    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:46.735643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.735612    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:46.735987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.735966    5420 patch_prober.go:29] interesting pod/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 13:33:46.736133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.736109    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 13:33:46.806749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:46.806716    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:47.047293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.015523    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:47.303180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.303091    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" event={"ID":"cf5edc82-7131-4b51-b078-ca321b6d1e16","Type":"ContainerStarted","Data":"76ea23fc100e66da057b5095cb6b23463d0abb5e03b911d0f1ae09ec42c1e8cb"}
Nov 03 13:33:47.303180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.303155    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" event={"ID":"cf5edc82-7131-4b51-b078-ca321b6d1e16","Type":"ContainerStarted","Data":"f82f836af664650f385fa03f88a50ccd60346f9f19450064f657e4a1b15ce87e"}
Nov 03 13:33:47.341060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.340998    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" podStartSLOduration=2.3409786280000002 podStartE2EDuration="2.340978628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:47.338069913 +0000 UTC m=+1468.415834940" watchObservedRunningTime="2026-11-03 13:33:47.340978628 +0000 UTC m=+1468.418743655"
Nov 03 13:33:47.516684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.516654    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:47.750802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.750754    5420 patch_prober.go:29] interesting pod/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 13:33:47.765015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.764972    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 13:33:47.765266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.764594    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:47.772233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.769314    5420 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 13:33:47.772233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:47.769364    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 13:33:48.103225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:48.103128    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:48.711502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:48.711465    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:48.735367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:48.733624    5420 patch_prober.go:29] interesting pod/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 13:33:48.735367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:48.733685    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 13:33:48.767238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:48.766661    5420 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 13:33:48.767238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:48.766720    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 13:33:49.457195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:49.449469    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:49.741299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:49.734584    5420 patch_prober.go:29] interesting pod/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 13:33:49.741299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:49.734646    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 13:33:49.769916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:49.765627    5420 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 13:33:49.769916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:49.765682    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 13:33:50.110282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:50.107988    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:50.743239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:50.742314    5420 patch_prober.go:29] interesting pod/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 13:33:50.743239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:50.742373    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 13:33:50.767243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:50.766754    5420 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 13:33:50.767243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:50.766806    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 13:33:50.805236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:50.800816    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:51.414888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:51.413532    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:51.740672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:51.740638    5420 patch_prober.go:29] interesting pod/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 13:33:51.740952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:51.740923    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 13:33:51.764561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:51.764527    5420 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 13:33:51.764948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:51.764586    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 13:33:52.267237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:52.266445    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:52.763733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:52.763700    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:52.764841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:52.764815    5420 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 13:33:52.765369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:52.765337    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 13:33:52.765515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:52.765500    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:52.765953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:52.765932    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:33:52.873234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:52.870089    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:53.278913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:53.278878    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:53.455682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:53.455647    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:53.770115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:53.770088    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:53.770626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:53.770606    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:53.771294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:53.771274    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:33:54.205395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:54.205346    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:54.731961 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:33:54.731928    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:33:55.038606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:55.038515    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:55.812076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:55.809023    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:56.408103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:56.408069    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:57.011340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:57.011306    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:57.725317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:57.725273    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:58.340137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:58.338028    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:59.059774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:59.059737    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:59.790064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:33:59.790031    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:00.320524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:00.320490    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:00.488515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:00.488487    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:00.628798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:00.628717    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:00.762973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:00.762930    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:34:00.830021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:00.829989    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:01.183646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:01.183618    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:01.654234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:01.653652    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:01.992671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:01.992642    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:02.413456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.413357    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:02.635331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.635283    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"]
Nov 03 13:34:02.641386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.641362    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.647876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.647853    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:34:02.649531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.649512    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:34:02.650281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.650255    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"]
Nov 03 13:34:02.735176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.735136    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:34:02.735686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.735267    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:34:02.750903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.750875    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zhs\" (UniqueName: \"kubernetes.io/projected/69dbdf79-1c7d-43d8-a73f-24225cecb025-kube-api-access-65zhs\") pod \"maas-api-revoke-sub-4ba22c82-rzzvq\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.751205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.751173    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba22c82-rzzvq\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.751471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.751452    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba22c82-rzzvq\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.852598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.852570    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba22c82-rzzvq\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.852750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.852652    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zhs\" (UniqueName: \"kubernetes.io/projected/69dbdf79-1c7d-43d8-a73f-24225cecb025-kube-api-access-65zhs\") pod \"maas-api-revoke-sub-4ba22c82-rzzvq\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.852750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.852735    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba22c82-rzzvq\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.853177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.853157    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba22c82-rzzvq\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.855175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.855152    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba22c82-rzzvq\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.862253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.862229    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zhs\" (UniqueName: \"kubernetes.io/projected/69dbdf79-1c7d-43d8-a73f-24225cecb025-kube-api-access-65zhs\") pod \"maas-api-revoke-sub-4ba22c82-rzzvq\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:02.969725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:02.969704    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:03.226077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:03.226045    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:03.263354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:03.263320    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"]
Nov 03 13:34:03.263747 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:34:03.263725    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dbdf79_1c7d_43d8_a73f_24225cecb025.slice/crio-6d6194ea7b136aeda896f38241976f5bacf8d2e437af1f551f3fedd7991fb295 WatchSource:0}: Error finding container 6d6194ea7b136aeda896f38241976f5bacf8d2e437af1f551f3fedd7991fb295: Status 404 returned error can't find the container with id 6d6194ea7b136aeda896f38241976f5bacf8d2e437af1f551f3fedd7991fb295
Nov 03 13:34:03.532189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:03.532120    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq" event={"ID":"69dbdf79-1c7d-43d8-a73f-24225cecb025","Type":"ContainerStarted","Data":"6d6194ea7b136aeda896f38241976f5bacf8d2e437af1f551f3fedd7991fb295"}
Nov 03 13:34:04.024231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:04.023829    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:04.553684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:04.553646    5420 generic.go:356] "Generic (PLEG): container finished" podID="69dbdf79-1c7d-43d8-a73f-24225cecb025" containerID="75a09119aa24c04529c2f5f9c03543fdf1ba33f9916875d0a1753e4df986b910" exitCode=0
Nov 03 13:34:04.553867 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:04.553719    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq" event={"ID":"69dbdf79-1c7d-43d8-a73f-24225cecb025","Type":"ContainerDied","Data":"75a09119aa24c04529c2f5f9c03543fdf1ba33f9916875d0a1753e4df986b910"}
Nov 03 13:34:04.618689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:04.618166    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:05.023758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.023718    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"]
Nov 03 13:34:05.028081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.028060    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a4eb40d-lwrt6"]
Nov 03 13:34:05.087655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.087627    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88edf335-9304-429c-8607-f2d8b7b446f5" path="/var/lib/kubelet/pods/88edf335-9304-429c-8607-f2d8b7b446f5/volumes"
Nov 03 13:34:05.467894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.467860    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:05.742782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.742758    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:05.902138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.902101    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-xks-ca\") pod \"69dbdf79-1c7d-43d8-a73f-24225cecb025\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") "
Nov 03 13:34:05.902375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.902189    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69dbdf79-1c7d-43d8-a73f-24225cecb025-kube-api-access-65zhs\" (UniqueName: \"kubernetes.io/projected/69dbdf79-1c7d-43d8-a73f-24225cecb025-kube-api-access-65zhs\") pod \"69dbdf79-1c7d-43d8-a73f-24225cecb025\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") "
Nov 03 13:34:05.902375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.902337    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-openshift-ca\") pod \"69dbdf79-1c7d-43d8-a73f-24225cecb025\" (UID: \"69dbdf79-1c7d-43d8-a73f-24225cecb025\") "
Nov 03 13:34:05.902700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.902676    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-openshift-ca" pod "69dbdf79-1c7d-43d8-a73f-24225cecb025" (UID: "69dbdf79-1c7d-43d8-a73f-24225cecb025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:05.903078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.903058    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:05.904244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.904206    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69dbdf79-1c7d-43d8-a73f-24225cecb025-kube-api-access-65zhs" pod "69dbdf79-1c7d-43d8-a73f-24225cecb025" (UID: "69dbdf79-1c7d-43d8-a73f-24225cecb025"). InnerVolumeSpecName "kube-api-access-65zhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:05.904244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:05.904203    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-xks-ca" pod "69dbdf79-1c7d-43d8-a73f-24225cecb025" (UID: "69dbdf79-1c7d-43d8-a73f-24225cecb025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:34:06.005015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:06.004949    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dbdf79-1c7d-43d8-a73f-24225cecb025-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:06.005015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:06.004979    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65zhs\" (UniqueName: \"kubernetes.io/projected/69dbdf79-1c7d-43d8-a73f-24225cecb025-kube-api-access-65zhs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:06.314937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:06.311917    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:06.575506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:06.575426    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq" event={"ID":"69dbdf79-1c7d-43d8-a73f-24225cecb025","Type":"ContainerDied","Data":"6d6194ea7b136aeda896f38241976f5bacf8d2e437af1f551f3fedd7991fb295"}
Nov 03 13:34:06.575506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:06.575495    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d6194ea7b136aeda896f38241976f5bacf8d2e437af1f551f3fedd7991fb295"
Nov 03 13:34:06.575688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:06.575443    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"
Nov 03 13:34:07.016836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:07.016786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:07.021016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:07.020988    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"]
Nov 03 13:34:07.183364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:07.183331    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:07.312392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:07.312309    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:07.901041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:07.901012    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:08.512455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:08.506123    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:09.047768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:09.047739    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:09.206562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:09.206531    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:09.738851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:09.738820    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:10.218089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:10.218059    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:11.016887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:11.016854    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"]
Nov 03 13:34:11.017266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:11.017199    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerName="istio-proxy" containerID="cri-o://83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2" gracePeriod=30
Nov 03 13:34:11.150922 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:11.150887    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:11.525176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:11.525138    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:12.022431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.022365    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76" podUID="8d139438-743f-4c7c-a0f3-1ded420d676b" containerName="payload-pre-processing" containerID="cri-o://1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215" gracePeriod=25
Nov 03 13:34:12.134418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.134368    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:12.344971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.344943    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:34:12.460546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.460512    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d139438-743f-4c7c-a0f3-1ded420d676b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d139438-743f-4c7c-a0f3-1ded420d676b-plugins-config-volume\") pod \"8d139438-743f-4c7c-a0f3-1ded420d676b\" (UID: \"8d139438-743f-4c7c-a0f3-1ded420d676b\") "
Nov 03 13:34:12.460797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.460777    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d139438-743f-4c7c-a0f3-1ded420d676b-kube-api-access-hlxj2\" (UniqueName: \"kubernetes.io/projected/8d139438-743f-4c7c-a0f3-1ded420d676b-kube-api-access-hlxj2\") pod \"8d139438-743f-4c7c-a0f3-1ded420d676b\" (UID: \"8d139438-743f-4c7c-a0f3-1ded420d676b\") "
Nov 03 13:34:12.461089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.461065    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d139438-743f-4c7c-a0f3-1ded420d676b-plugins-config-volume" pod "8d139438-743f-4c7c-a0f3-1ded420d676b" (UID: "8d139438-743f-4c7c-a0f3-1ded420d676b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:12.461611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.461557    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d139438-743f-4c7c-a0f3-1ded420d676b-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:12.463653 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.463620    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d139438-743f-4c7c-a0f3-1ded420d676b-kube-api-access-hlxj2" pod "8d139438-743f-4c7c-a0f3-1ded420d676b" (UID: "8d139438-743f-4c7c-a0f3-1ded420d676b"). InnerVolumeSpecName "kube-api-access-hlxj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:12.562937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.562870    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlxj2\" (UniqueName: \"kubernetes.io/projected/8d139438-743f-4c7c-a0f3-1ded420d676b-kube-api-access-hlxj2\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:12.635061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.635032    5420 generic.go:356] "Generic (PLEG): container finished" podID="8d139438-743f-4c7c-a0f3-1ded420d676b" containerID="1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215" exitCode=0
Nov 03 13:34:12.635237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.635075    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76" event={"ID":"8d139438-743f-4c7c-a0f3-1ded420d676b","Type":"ContainerDied","Data":"1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215"}
Nov 03 13:34:12.635237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.635081    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"
Nov 03 13:34:12.635237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.635113    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76" event={"ID":"8d139438-743f-4c7c-a0f3-1ded420d676b","Type":"ContainerDied","Data":"ad050b4bbb56232b9be3288869f53e9e797e5489cee57cf0eae1d7c46dddf5b6"}
Nov 03 13:34:12.635237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.635142    5420 scope.go:122] "RemoveContainer" containerID="1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215"
Nov 03 13:34:12.647173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.647156    5420 scope.go:122] "RemoveContainer" containerID="1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215"
Nov 03 13:34:12.647456 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:12.647437    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215\": container with ID starting with 1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215 not found: ID does not exist" containerID="1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215"
Nov 03 13:34:12.647520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.647464    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215"} err="failed to get container status \"1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215\": rpc error: code = NotFound desc = could not find container \"1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215\": container with ID starting with 1d8525c45ff6e55b9ca51002653b2762f764e86d0640d6aad5db78699a560215 not found: ID does not exist"
Nov 03 13:34:12.662611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.662591    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"]
Nov 03 13:34:12.669395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.669374    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-418af8-885c454d7-mkx76"]
Nov 03 13:34:12.879272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:12.879188    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:13.031701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:13.031664    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:13.086969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:13.086939    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d139438-743f-4c7c-a0f3-1ded420d676b" path="/var/lib/kubelet/pods/8d139438-743f-4c7c-a0f3-1ded420d676b/volumes"
Nov 03 13:34:13.528934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:13.527178    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:14.042304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.042274    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:14.163139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.163107    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"]
Nov 03 13:34:14.163355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.163330    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy" containerID="cri-o://a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0" gracePeriod=30
Nov 03 13:34:14.167820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.167795    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"]
Nov 03 13:34:14.168013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.167991    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy" containerID="cri-o://08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1" gracePeriod=30
Nov 03 13:34:14.189306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.189278    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"]
Nov 03 13:34:14.189494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.189473    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerName="istio-proxy" containerID="cri-o://22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d" gracePeriod=30
Nov 03 13:34:14.197990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.197965    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"]
Nov 03 13:34:14.198188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.198166    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy" containerID="cri-o://fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f" gracePeriod=30
Nov 03 13:34:14.293599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.293509    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:14.945765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:14.941943    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:15.030119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:15.030086    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"]
Nov 03 13:34:15.036606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:15.036570    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e21aca-qpgtq"]
Nov 03 13:34:15.086148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:15.086114    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b" path="/var/lib/kubelet/pods/7a3fcace-0be4-4c8b-ac35-f83cf8f25a4b/volumes"
Nov 03 13:34:15.465729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:15.465687    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:15.612291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:15.612257    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:15.918122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:15.918019    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:16.086685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.086648    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:16.301970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.301942    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:34:16.327554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.325293    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"]
Nov 03 13:34:16.327554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.325517    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy" containerID="cri-o://036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0" gracePeriod=30
Nov 03 13:34:16.424779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.424739    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:16.425608 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.425586    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-kube-api-access-cnd4z\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-kube-api-access-cnd4z\") pod \"115dd85d-bf72-4d2b-819d-84c66d23403b\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") "
Nov 03 13:34:16.425725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.425632    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-data\") pod \"115dd85d-bf72-4d2b-819d-84c66d23403b\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") "
Nov 03 13:34:16.425725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.425663    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-podinfo\") pod \"115dd85d-bf72-4d2b-819d-84c66d23403b\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") "
Nov 03 13:34:16.425725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.425712    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-envoy\") pod \"115dd85d-bf72-4d2b-819d-84c66d23403b\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") "
Nov 03 13:34:16.425877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.425749    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-token\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-token\") pod \"115dd85d-bf72-4d2b-819d-84c66d23403b\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") "
Nov 03 13:34:16.425935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.425884    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115dd85d-bf72-4d2b-819d-84c66d23403b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115dd85d-bf72-4d2b-819d-84c66d23403b-istiod-ca-cert\") pod \"115dd85d-bf72-4d2b-819d-84c66d23403b\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") "
Nov 03 13:34:16.425997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.425953    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-data" pod "115dd85d-bf72-4d2b-819d-84c66d23403b" (UID: "115dd85d-bf72-4d2b-819d-84c66d23403b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.426058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.426011    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-certs\") pod \"115dd85d-bf72-4d2b-819d-84c66d23403b\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") "
Nov 03 13:34:16.426514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.426189    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-credential-socket\") pod \"115dd85d-bf72-4d2b-819d-84c66d23403b\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") "
Nov 03 13:34:16.426514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.426247    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-certs" pod "115dd85d-bf72-4d2b-819d-84c66d23403b" (UID: "115dd85d-bf72-4d2b-819d-84c66d23403b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.426514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.426332    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115dd85d-bf72-4d2b-819d-84c66d23403b-istiod-ca-cert" pod "115dd85d-bf72-4d2b-819d-84c66d23403b" (UID: "115dd85d-bf72-4d2b-819d-84c66d23403b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:16.426514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.426429    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-socket\") pod \"115dd85d-bf72-4d2b-819d-84c66d23403b\" (UID: \"115dd85d-bf72-4d2b-819d-84c66d23403b\") "
Nov 03 13:34:16.426514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.426438    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-credential-socket" pod "115dd85d-bf72-4d2b-819d-84c66d23403b" (UID: "115dd85d-bf72-4d2b-819d-84c66d23403b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.426950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.426775    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-socket" pod "115dd85d-bf72-4d2b-819d-84c66d23403b" (UID: "115dd85d-bf72-4d2b-819d-84c66d23403b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.427361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.427338    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115dd85d-bf72-4d2b-819d-84c66d23403b-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.427506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.427365    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.427506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.427380    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.427506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.427394    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.427506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.427409    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.428142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.428122    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-podinfo" pod "115dd85d-bf72-4d2b-819d-84c66d23403b" (UID: "115dd85d-bf72-4d2b-819d-84c66d23403b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:16.428398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.428374    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-kube-api-access-cnd4z" pod "115dd85d-bf72-4d2b-819d-84c66d23403b" (UID: "115dd85d-bf72-4d2b-819d-84c66d23403b"). InnerVolumeSpecName "kube-api-access-cnd4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:16.428477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.428402    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-envoy" pod "115dd85d-bf72-4d2b-819d-84c66d23403b" (UID: "115dd85d-bf72-4d2b-819d-84c66d23403b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:16.428521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.428488    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-token" pod "115dd85d-bf72-4d2b-819d-84c66d23403b" (UID: "115dd85d-bf72-4d2b-819d-84c66d23403b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:16.548140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.544204    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.548140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.544257    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.548140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.544270    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnd4z\" (UniqueName: \"kubernetes.io/projected/115dd85d-bf72-4d2b-819d-84c66d23403b-kube-api-access-cnd4z\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.548140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.544285    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115dd85d-bf72-4d2b-819d-84c66d23403b-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:16.649238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.649104    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:16.695285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.694318    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"
Nov 03 13:34:16.695285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.695092    5420 generic.go:356] "Generic (PLEG): container finished" podID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerID="83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2" exitCode=0
Nov 03 13:34:16.703232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.696269    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" event={"ID":"115dd85d-bf72-4d2b-819d-84c66d23403b","Type":"ContainerDied","Data":"83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2"}
Nov 03 13:34:16.703232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.696341    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt" event={"ID":"115dd85d-bf72-4d2b-819d-84c66d23403b","Type":"ContainerDied","Data":"6d34dff24de1b074ba32ac46d3e6a5a0f0f64be6a23e59c155650d997cb4abb5"}
Nov 03 13:34:16.703232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.696383    5420 scope.go:122] "RemoveContainer" containerID="83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2"
Nov 03 13:34:16.715092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.715071    5420 scope.go:122] "RemoveContainer" containerID="83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2"
Nov 03 13:34:16.716443 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:16.716419    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2\": container with ID starting with 83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2 not found: ID does not exist" containerID="83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2"
Nov 03 13:34:16.716551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.716452    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2"} err="failed to get container status \"83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2\": rpc error: code = NotFound desc = could not find container \"83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2\": container with ID starting with 83d4095835602c0a34e79c4b41748435203e9d8a6215d8c28cb00906f8021cc2 not found: ID does not exist"
Nov 03 13:34:16.729903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.729872    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"]
Nov 03 13:34:16.734988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:16.734962    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-vx4dt"]
Nov 03 13:34:17.087945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:17.087915    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" path="/var/lib/kubelet/pods/115dd85d-bf72-4d2b-819d-84c66d23403b/volumes"
Nov 03 13:34:17.209630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:17.209598    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:17.440354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:17.428204    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:17.707235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:17.705740    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:17.993875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:17.993799    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:18.237514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:18.233746    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:18.502835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:18.502805    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:19.097192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.097157    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:19.490902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.490875    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:34:19.506361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.506334    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:34:19.541933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.541908    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:34:19.566155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.566134    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:34:19.645957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.645509    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istiod-ca-cert\") pod \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") "
Nov 03 13:34:19.645957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.645578    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-certs\") pod \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") "
Nov 03 13:34:19.645957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.645616    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-token\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-token\") pod \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") "
Nov 03 13:34:19.645957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.645693    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-token\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-token\") pod \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") "
Nov 03 13:34:19.645957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.645742    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-podinfo\") pod \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") "
Nov 03 13:34:19.647035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.645851    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-data\") pod \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") "
Nov 03 13:34:19.647164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647116    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-socket\") pod \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") "
Nov 03 13:34:19.647164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.645963    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istiod-ca-cert" pod "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" (UID: "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:19.647363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.646578    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-data" pod "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" (UID: "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.647363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647180    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-socket\") pod \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") "
Nov 03 13:34:19.647363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647235    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-credential-socket\") pod \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") "
Nov 03 13:34:19.647363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647263    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-envoy\") pod \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") "
Nov 03 13:34:19.647363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647291    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-certs\") pod \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") "
Nov 03 13:34:19.647718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647364    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-data\") pod \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") "
Nov 03 13:34:19.647718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647441    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-kube-api-access-2mznf\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-kube-api-access-2mznf\") pod \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\" (UID: \"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9\") "
Nov 03 13:34:19.647718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647474    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istiod-ca-cert\") pod \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") "
Nov 03 13:34:19.647718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647529    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-envoy\") pod \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") "
Nov 03 13:34:19.647718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.646829    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-certs" pod "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" (UID: "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.647718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647413    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-socket" pod "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" (UID: "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.647718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647564    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-kube-api-access-8xg9k\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-kube-api-access-8xg9k\") pod \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") "
Nov 03 13:34:19.647718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647640    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-credential-socket\") pod \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") "
Nov 03 13:34:19.647718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647688    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-podinfo\") pod \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\" (UID: \"eb1e78d4-0bff-4405-9e3a-a11d0835a719\") "
Nov 03 13:34:19.648291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647725    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-credential-socket" pod "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" (UID: "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.648291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.647991    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-data" pod "eb1e78d4-0bff-4405-9e3a-a11d0835a719" (UID: "eb1e78d4-0bff-4405-9e3a-a11d0835a719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.648932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.648596    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.648932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.648623    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.648932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.648639    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.648932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.648655    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.648932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.648669    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.648932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.648684    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.652694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.652531    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-envoy" pod "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" (UID: "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.654240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.653929    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-socket" pod "eb1e78d4-0bff-4405-9e3a-a11d0835a719" (UID: "eb1e78d4-0bff-4405-9e3a-a11d0835a719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.654240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.654051    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-token" pod "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" (UID: "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:19.654424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.654404    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-credential-socket" pod "eb1e78d4-0bff-4405-9e3a-a11d0835a719" (UID: "eb1e78d4-0bff-4405-9e3a-a11d0835a719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.654607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.654563    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-certs" pod "eb1e78d4-0bff-4405-9e3a-a11d0835a719" (UID: "eb1e78d4-0bff-4405-9e3a-a11d0835a719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.654728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.654708    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istiod-ca-cert" pod "eb1e78d4-0bff-4405-9e3a-a11d0835a719" (UID: "eb1e78d4-0bff-4405-9e3a-a11d0835a719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:19.656430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.656385    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-kube-api-access-2mznf" pod "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" (UID: "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9"). InnerVolumeSpecName "kube-api-access-2mznf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:19.657934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.657906    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-kube-api-access-8xg9k" pod "eb1e78d4-0bff-4405-9e3a-a11d0835a719" (UID: "eb1e78d4-0bff-4405-9e3a-a11d0835a719"). InnerVolumeSpecName "kube-api-access-8xg9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:19.661643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.661564    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-envoy" pod "eb1e78d4-0bff-4405-9e3a-a11d0835a719" (UID: "eb1e78d4-0bff-4405-9e3a-a11d0835a719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.661643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.661581    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-token" pod "eb1e78d4-0bff-4405-9e3a-a11d0835a719" (UID: "eb1e78d4-0bff-4405-9e3a-a11d0835a719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:19.661772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.661662    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-podinfo" pod "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" (UID: "6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:19.663685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.663662    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-podinfo" pod "eb1e78d4-0bff-4405-9e3a-a11d0835a719" (UID: "eb1e78d4-0bff-4405-9e3a-a11d0835a719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:19.671322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.671294    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:34:19.673374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.673352    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:34:19.749598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749570    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-certs\") pod \"2c74d124-2b67-4752-9725-ef7146010303\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") "
Nov 03 13:34:19.749598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749605    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istiod-ca-cert\") pod \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") "
Nov 03 13:34:19.749839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749641    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-socket\") pod \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") "
Nov 03 13:34:19.749839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749724    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-certs\") pod \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") "
Nov 03 13:34:19.749839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749751    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-data\") pod \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") "
Nov 03 13:34:19.749839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749783    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-envoy\") pod \"2c74d124-2b67-4752-9725-ef7146010303\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") "
Nov 03 13:34:19.749839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749828    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c74d124-2b67-4752-9725-ef7146010303-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c74d124-2b67-4752-9725-ef7146010303-istiod-ca-cert\") pod \"2c74d124-2b67-4752-9725-ef7146010303\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") "
Nov 03 13:34:19.750122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749858    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-kube-api-access-b889q\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-kube-api-access-b889q\") pod \"2c74d124-2b67-4752-9725-ef7146010303\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") "
Nov 03 13:34:19.750122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749892    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-istio-token\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-istio-token\") pod \"2c74d124-2b67-4752-9725-ef7146010303\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") "
Nov 03 13:34:19.750122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749884    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-certs" pod "2c74d124-2b67-4752-9725-ef7146010303" (UID: "2c74d124-2b67-4752-9725-ef7146010303"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.750122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.749984    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-socket" pod "777a0f06-59b2-42ba-b18f-77c6c351a7cd" (UID: "777a0f06-59b2-42ba-b18f-77c6c351a7cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.750122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750042    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-data\") pod \"2c74d124-2b67-4752-9725-ef7146010303\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") "
Nov 03 13:34:19.750122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750058    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istiod-ca-cert" pod "777a0f06-59b2-42ba-b18f-77c6c351a7cd" (UID: "777a0f06-59b2-42ba-b18f-77c6c351a7cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:19.750122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750080    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c74d124-2b67-4752-9725-ef7146010303-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c74d124-2b67-4752-9725-ef7146010303-istio-podinfo\") pod \"2c74d124-2b67-4752-9725-ef7146010303\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") "
Nov 03 13:34:19.750576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750135    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-credential-socket\") pod \"2c74d124-2b67-4752-9725-ef7146010303\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") "
Nov 03 13:34:19.750576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750169    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-socket\") pod \"2c74d124-2b67-4752-9725-ef7146010303\" (UID: \"2c74d124-2b67-4752-9725-ef7146010303\") "
Nov 03 13:34:19.750576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750255    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-credential-socket\") pod \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") "
Nov 03 13:34:19.750576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750287    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-envoy\") pod \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") "
Nov 03 13:34:19.750576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750324    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-kube-api-access-mhsbw\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-kube-api-access-mhsbw\") pod \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") "
Nov 03 13:34:19.750576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750345    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-data" pod "777a0f06-59b2-42ba-b18f-77c6c351a7cd" (UID: "777a0f06-59b2-42ba-b18f-77c6c351a7cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.750576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750361    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-token\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-token\") pod \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") "
Nov 03 13:34:19.750576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750405    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-podinfo\") pod \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\" (UID: \"777a0f06-59b2-42ba-b18f-77c6c351a7cd\") "
Nov 03 13:34:19.750576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.750516    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-certs" pod "777a0f06-59b2-42ba-b18f-77c6c351a7cd" (UID: "777a0f06-59b2-42ba-b18f-77c6c351a7cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751323    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-credential-socket" pod "777a0f06-59b2-42ba-b18f-77c6c351a7cd" (UID: "777a0f06-59b2-42ba-b18f-77c6c351a7cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751361    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mznf\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-kube-api-access-2mznf\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751532    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-credential-socket" pod "2c74d124-2b67-4752-9725-ef7146010303" (UID: "2c74d124-2b67-4752-9725-ef7146010303"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751720    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-socket" pod "2c74d124-2b67-4752-9725-ef7146010303" (UID: "2c74d124-2b67-4752-9725-ef7146010303"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751747    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751770    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751786    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xg9k\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-kube-api-access-8xg9k\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751802    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751816    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751829    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751843    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751857    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751871    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751885    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751899    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751913    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1e78d4-0bff-4405-9e3a-a11d0835a719-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751926    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751940    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751954    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.751969    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1e78d4-0bff-4405-9e3a-a11d0835a719-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.752345    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c74d124-2b67-4752-9725-ef7146010303-istiod-ca-cert" pod "2c74d124-2b67-4752-9725-ef7146010303" (UID: "2c74d124-2b67-4752-9725-ef7146010303"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.752507    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-envoy" pod "2c74d124-2b67-4752-9725-ef7146010303" (UID: "2c74d124-2b67-4752-9725-ef7146010303"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.753706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.752859    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-data" pod "2c74d124-2b67-4752-9725-ef7146010303" (UID: "2c74d124-2b67-4752-9725-ef7146010303"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.755000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.753886    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-kube-api-access-b889q" pod "2c74d124-2b67-4752-9725-ef7146010303" (UID: "2c74d124-2b67-4752-9725-ef7146010303"). InnerVolumeSpecName "kube-api-access-b889q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:19.755000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.754967    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-podinfo" pod "777a0f06-59b2-42ba-b18f-77c6c351a7cd" (UID: "777a0f06-59b2-42ba-b18f-77c6c351a7cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:19.755111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.755063    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-envoy" pod "777a0f06-59b2-42ba-b18f-77c6c351a7cd" (UID: "777a0f06-59b2-42ba-b18f-77c6c351a7cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:19.755307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.755284    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-token" pod "777a0f06-59b2-42ba-b18f-77c6c351a7cd" (UID: "777a0f06-59b2-42ba-b18f-77c6c351a7cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:19.755622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.755593    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-istio-token" pod "2c74d124-2b67-4752-9725-ef7146010303" (UID: "2c74d124-2b67-4752-9725-ef7146010303"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:19.755622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.755609    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c74d124-2b67-4752-9725-ef7146010303-istio-podinfo" pod "2c74d124-2b67-4752-9725-ef7146010303" (UID: "2c74d124-2b67-4752-9725-ef7146010303"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:19.757352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.757325    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-kube-api-access-mhsbw" pod "777a0f06-59b2-42ba-b18f-77c6c351a7cd" (UID: "777a0f06-59b2-42ba-b18f-77c6c351a7cd"). InnerVolumeSpecName "kube-api-access-mhsbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:19.813889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.813854    5420 generic.go:356] "Generic (PLEG): container finished" podID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerID="08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1" exitCode=0
Nov 03 13:34:19.815001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.814985    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"
Nov 03 13:34:19.815337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.815280    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" event={"ID":"eb1e78d4-0bff-4405-9e3a-a11d0835a719","Type":"ContainerDied","Data":"08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1"}
Nov 03 13:34:19.815666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.815646    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt" event={"ID":"eb1e78d4-0bff-4405-9e3a-a11d0835a719","Type":"ContainerDied","Data":"0404c620f83f8cbe47270ea4a011bd30d06c3308746b98490c447eb90a9463f4"}
Nov 03 13:34:19.815811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.815690    5420 scope.go:122] "RemoveContainer" containerID="08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1"
Nov 03 13:34:19.817146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.817122    5420 generic.go:356] "Generic (PLEG): container finished" podID="2c74d124-2b67-4752-9725-ef7146010303" containerID="a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0" exitCode=0
Nov 03 13:34:19.817384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.817294    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" event={"ID":"2c74d124-2b67-4752-9725-ef7146010303","Type":"ContainerDied","Data":"a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0"}
Nov 03 13:34:19.817384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.817342    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7" event={"ID":"2c74d124-2b67-4752-9725-ef7146010303","Type":"ContainerDied","Data":"1624026e3a17578ddef1fffe18d121464a9354c6c8c1ca6e8a3bd052d52bd2e9"}
Nov 03 13:34:19.817974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.817954    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"
Nov 03 13:34:19.819663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.819640    5420 generic.go:356] "Generic (PLEG): container finished" podID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerID="fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f" exitCode=0
Nov 03 13:34:19.819771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.819674    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"
Nov 03 13:34:19.819771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.819745    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" event={"ID":"777a0f06-59b2-42ba-b18f-77c6c351a7cd","Type":"ContainerDied","Data":"fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f"}
Nov 03 13:34:19.819934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.819798    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k" event={"ID":"777a0f06-59b2-42ba-b18f-77c6c351a7cd","Type":"ContainerDied","Data":"ae05f34def2f0b4cda8d4d1182c50dc45586eb9fceb755321e3c9fac45f9b3d1"}
Nov 03 13:34:19.822195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.822171    5420 generic.go:356] "Generic (PLEG): container finished" podID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerID="22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d" exitCode=0
Nov 03 13:34:19.822313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.822265    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" event={"ID":"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9","Type":"ContainerDied","Data":"22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d"}
Nov 03 13:34:19.822313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.822301    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq" event={"ID":"6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9","Type":"ContainerDied","Data":"0e86b92de249cfa612a5db8a7ce0b770dd25bd952f42758fde229f6515b71c33"}
Nov 03 13:34:19.822446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.822364    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"
Nov 03 13:34:19.839120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.838995    5420 scope.go:122] "RemoveContainer" containerID="08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1"
Nov 03 13:34:19.840441 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:19.840295    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1\": container with ID starting with 08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1 not found: ID does not exist" containerID="08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1"
Nov 03 13:34:19.840441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.840333    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1"} err="failed to get container status \"08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1\": rpc error: code = NotFound desc = could not find container \"08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1\": container with ID starting with 08e81428fb65d3d3510e49d4bb903e8b81eb35178b7376e927466f202c0292b1 not found: ID does not exist"
Nov 03 13:34:19.840441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.840356    5420 scope.go:122] "RemoveContainer" containerID="a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0"
Nov 03 13:34:19.851164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.851149    5420 scope.go:122] "RemoveContainer" containerID="a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0"
Nov 03 13:34:19.852282 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:19.852194    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0\": container with ID starting with a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0 not found: ID does not exist" containerID="a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0"
Nov 03 13:34:19.852377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.852297    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0"} err="failed to get container status \"a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0\": rpc error: code = NotFound desc = could not find container \"a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0\": container with ID starting with a720b1353aba14652766557573bd0b35f7244fae455f39733901c66361ef24f0 not found: ID does not exist"
Nov 03 13:34:19.852377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.852321    5420 scope.go:122] "RemoveContainer" containerID="fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f"
Nov 03 13:34:19.853077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853059    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853078    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c74d124-2b67-4752-9725-ef7146010303-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853088    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853096    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853110    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853122    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853131    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhsbw\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-kube-api-access-mhsbw\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853143    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853155    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/777a0f06-59b2-42ba-b18f-77c6c351a7cd-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853165    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c74d124-2b67-4752-9725-ef7146010303-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853173    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c74d124-2b67-4752-9725-ef7146010303-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853182    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b889q\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-kube-api-access-b889q\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.853543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.853190    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c74d124-2b67-4752-9725-ef7146010303-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:19.862298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.862199    5420 scope.go:122] "RemoveContainer" containerID="fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f"
Nov 03 13:34:19.862624 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:19.862578    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f\": container with ID starting with fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f not found: ID does not exist" containerID="fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f"
Nov 03 13:34:19.862624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.862610    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f"} err="failed to get container status \"fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f\": rpc error: code = NotFound desc = could not find container \"fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f\": container with ID starting with fce1cd59c00c05bb882894e0b7503aa53a8a8837f852e3f960afc9b0aae4aa6f not found: ID does not exist"
Nov 03 13:34:19.862804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.862633    5420 scope.go:122] "RemoveContainer" containerID="22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d"
Nov 03 13:34:19.875262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.875229    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"]
Nov 03 13:34:19.883224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.883183    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-bhq2k"]
Nov 03 13:34:19.888603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.888584    5420 scope.go:122] "RemoveContainer" containerID="22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d"
Nov 03 13:34:19.888891 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:19.888870    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d\": container with ID starting with 22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d not found: ID does not exist" containerID="22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d"
Nov 03 13:34:19.888966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.888903    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d"} err="failed to get container status \"22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d\": rpc error: code = NotFound desc = could not find container \"22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d\": container with ID starting with 22b0fa13b14f5367ab8b36796733c054f800c1ee608c17b5aa1a066041d7e50d not found: ID does not exist"
Nov 03 13:34:19.896088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.896038    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"]
Nov 03 13:34:19.910251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.910229    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-nbkv7"]
Nov 03 13:34:19.935042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.935020    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"]
Nov 03 13:34:19.977546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.977507    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-zv7lt"]
Nov 03 13:34:19.980052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.980011    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:19.998756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:19.998724    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"]
Nov 03 13:34:20.004750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:20.004714    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-ggzrq"]
Nov 03 13:34:20.757070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:20.757038    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:21.085963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.085877    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c74d124-2b67-4752-9725-ef7146010303" path="/var/lib/kubelet/pods/2c74d124-2b67-4752-9725-ef7146010303/volumes"
Nov 03 13:34:21.086518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.086495    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" path="/var/lib/kubelet/pods/6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9/volumes"
Nov 03 13:34:21.087029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.087009    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" path="/var/lib/kubelet/pods/777a0f06-59b2-42ba-b18f-77c6c351a7cd/volumes"
Nov 03 13:34:21.087885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.087862    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" path="/var/lib/kubelet/pods/eb1e78d4-0bff-4405-9e3a-a11d0835a719/volumes"
Nov 03 13:34:21.268867 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.268833    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:21.505997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.504349    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:21.568244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.567897    5420 scope.go:122] "RemoveContainer" containerID="88c48efd6a19c1b90bd47d2fb84bee0d59b1f2a6222e325eda708dbbb73c5353"
Nov 03 13:34:21.591832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.591805    5420 scope.go:122] "RemoveContainer" containerID="8310b59a1de1c8eb922ce5e3f04e882d173331f3a46eaf3281e3b0c108c82d13"
Nov 03 13:34:21.603384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.603353    5420 scope.go:122] "RemoveContainer" containerID="a657a449c789666c432852f5b741167d9af09d4c18c6a708ea81fbb33199108c"
Nov 03 13:34:21.617682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.617657    5420 scope.go:122] "RemoveContainer" containerID="74ffff3291d803106b58f56b32f923e83a7210b4eeab25714ceae665f3ce4d37"
Nov 03 13:34:21.627778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.627757    5420 scope.go:122] "RemoveContainer" containerID="86eb11f58a9801244af9966b816e9740f0815bfaa3fd48c519722b4d098561b5"
Nov 03 13:34:21.672965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.672946    5420 scope.go:122] "RemoveContainer" containerID="ffd480ebb0d618888cadc9b7ebacc43672976bd01b5accad0cdd20d325efc24b"
Nov 03 13:34:21.677334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.677316    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:34:21.682378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.682363    5420 scope.go:122] "RemoveContainer" containerID="769b1cfde4ed98d82c02291f133ddc1b36593d74090a2ad21e8385b9dfc85b57"
Nov 03 13:34:21.691425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.691409    5420 scope.go:122] "RemoveContainer" containerID="cd3fc9b8327e0e4cd02e23a45b214610f904c91630eb0d8b2f39c4b9839da558"
Nov 03 13:34:21.806496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806473    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-certs\") pod \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") "
Nov 03 13:34:21.806918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806553    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-credential-socket\") pod \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") "
Nov 03 13:34:21.806918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806592    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-data\") pod \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") "
Nov 03 13:34:21.806918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806638    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-socket\") pod \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") "
Nov 03 13:34:21.806918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806718    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-kube-api-access-nqmlq\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-kube-api-access-nqmlq\") pod \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") "
Nov 03 13:34:21.806918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806764    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-envoy\") pod \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") "
Nov 03 13:34:21.806918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806796    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-token\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-token\") pod \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") "
Nov 03 13:34:21.806918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806760    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-certs" pod "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" (UID: "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:21.806918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806857    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-podinfo\") pod \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") "
Nov 03 13:34:21.806918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806872    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-credential-socket" pod "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" (UID: "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:21.807445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806968    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istiod-ca-cert\") pod \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\" (UID: \"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710\") "
Nov 03 13:34:21.807445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.806998    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-socket" pod "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" (UID: "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:21.807445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.807280    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-data" pod "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" (UID: "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:21.807445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.807425    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istiod-ca-cert" pod "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" (UID: "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:21.807811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.807792    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:21.807883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.807819    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:21.807883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.807834    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:21.807883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.807849    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:21.807883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.807863    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:21.808945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.808914    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-envoy" pod "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" (UID: "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:34:21.809098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.809074    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-token" pod "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" (UID: "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:21.809312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.809293    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-podinfo" pod "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" (UID: "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:34:21.809402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.809383    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-kube-api-access-nqmlq" pod "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" (UID: "79cc5a21-b6ec-49b4-8d77-51fa0a5a1710"). InnerVolumeSpecName "kube-api-access-nqmlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:21.849724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.849693    5420 generic.go:356] "Generic (PLEG): container finished" podID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerID="036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0" exitCode=0
Nov 03 13:34:21.849869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.849757    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"
Nov 03 13:34:21.849869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.849772    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" event={"ID":"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710","Type":"ContainerDied","Data":"036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0"}
Nov 03 13:34:21.849869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.849826    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97" event={"ID":"79cc5a21-b6ec-49b4-8d77-51fa0a5a1710","Type":"ContainerDied","Data":"cf12048ffffdae121709e25efd67d6359f07e1694bc207d66cb6769c7ddd517f"}
Nov 03 13:34:21.849869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.849865    5420 scope.go:122] "RemoveContainer" containerID="036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0"
Nov 03 13:34:21.861534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.861517    5420 scope.go:122] "RemoveContainer" containerID="8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103"
Nov 03 13:34:21.872129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.872109    5420 scope.go:122] "RemoveContainer" containerID="036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0"
Nov 03 13:34:21.873167 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:21.872617    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0\": container with ID starting with 036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0 not found: ID does not exist" containerID="036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0"
Nov 03 13:34:21.873167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.872656    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0"} err="failed to get container status \"036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0\": rpc error: code = NotFound desc = could not find container \"036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0\": container with ID starting with 036465a47930a22f785dcca1fa5f65c3a45e0d050a652f71fa0d06d67bb488d0 not found: ID does not exist"
Nov 03 13:34:21.873167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.872684    5420 scope.go:122] "RemoveContainer" containerID="8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103"
Nov 03 13:34:21.873897 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:21.873872    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103\": container with ID starting with 8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103 not found: ID does not exist" containerID="8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103"
Nov 03 13:34:21.873979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.873906    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103"} err="failed to get container status \"8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103\": rpc error: code = NotFound desc = could not find container \"8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103\": container with ID starting with 8ca4f50d5141aa0d781d4287d353676a42381b8ce6b9925f7f525f6916e22103 not found: ID does not exist"
Nov 03 13:34:21.878581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.878562    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"]
Nov 03 13:34:21.884617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.884594    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-418af8-openshift-default-564fb5dc9f-l9k97"]
Nov 03 13:34:21.909552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.909525    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqmlq\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-kube-api-access-nqmlq\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:21.909642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.909555    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:21.909642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.909571    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:21.909642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:21.909585    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:22.198170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:22.198096    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:22.369918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:22.369880    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:22.535660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:22.535618    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:22.709765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:22.709735    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:23.001172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001141    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"]
Nov 03 13:34:23.001839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001824    5420 cpu_manager.go:414] "containerMap: removing container" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy"
Nov 03 13:34:23.001839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001839    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy"
Nov 03 13:34:23.001933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001852    5420 cpu_manager.go:414] "containerMap: removing container" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy"
Nov 03 13:34:23.001933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001858    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy"
Nov 03 13:34:23.001933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001868    5420 cpu_manager.go:414] "containerMap: removing container" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerName="istio-proxy"
Nov 03 13:34:23.001933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001873    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerName="istio-proxy"
Nov 03 13:34:23.001933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001888    5420 cpu_manager.go:414] "containerMap: removing container" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy"
Nov 03 13:34:23.001933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001893    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy"
Nov 03 13:34:23.001933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001919    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8d139438-743f-4c7c-a0f3-1ded420d676b" containerName="payload-pre-processing"
Nov 03 13:34:23.001933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001924    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d139438-743f-4c7c-a0f3-1ded420d676b" containerName="payload-pre-processing"
Nov 03 13:34:23.001933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001937    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001941    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001951    5420 cpu_manager.go:414] "containerMap: removing container" podUID="69dbdf79-1c7d-43d8-a73f-24225cecb025" containerName="revoke-keys"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001955    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69dbdf79-1c7d-43d8-a73f-24225cecb025" containerName="revoke-keys"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001963    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.001967    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.002053    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.002062    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d139438-743f-4c7c-a0f3-1ded420d676b" containerName="payload-pre-processing"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.002069    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c74d124-2b67-4752-9725-ef7146010303" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.002074    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="69dbdf79-1c7d-43d8-a73f-24225cecb025" containerName="revoke-keys"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.002081    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba49e15-e0aa-4364-aa9a-cfe7dc1f88d9" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.002088    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1e78d4-0bff-4405-9e3a-a11d0835a719" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.002094    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.002100    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="115dd85d-bf72-4d2b-819d-84c66d23403b" containerName="istio-proxy"
Nov 03 13:34:23.002203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.002106    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="777a0f06-59b2-42ba-b18f-77c6c351a7cd" containerName="istio-proxy"
Nov 03 13:34:23.007453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.007436    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.017010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.016989    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"]
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019148    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"openshift-service-ca.crt\""
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019179    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"kube-root-ca.crt\""
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019200    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a62d22\""
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019229    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-ec4cc2d7b734\""
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019265    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"routing-overlay\""
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019453    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-4d57ed2d3b27\""
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019457    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-c03d3bf9f10f\""
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019532    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-45edb24b5c7b\""
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019636    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"payload-processing-external-model-e2e--f52a173c-dockercfg-d5wkp\""
Nov 03 13:34:23.021254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.019649    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"e2e-cfl-e87c5e-provider-cred\""
Nov 03 13:34:23.085883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.085854    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" path="/var/lib/kubelet/pods/79cc5a21-b6ec-49b4-8d77-51fa0a5a1710/volumes"
Nov 03 13:34:23.125884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.125858    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.125990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.125971    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.126045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.126000    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdhj\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-kube-api-access-wkdhj\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.126106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.126098    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.227578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.227550    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.227698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.227683    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.227845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.227827    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.227900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.227878    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdhj\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-kube-api-access-wkdhj\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.228564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.228539    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.228659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.228606    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.230461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.230445    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.237897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.237876    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdhj\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-kube-api-access-wkdhj\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.325272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.325198    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.613855 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:34:23.613825    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3425a2_8dc5_4666_bf51_b52abeb43601.slice/crio-4e01b6f66f6ca70c802e20766914f86137ddb4a707fc88b10e44a01b6d4ff2b3 WatchSource:0}: Error finding container 4e01b6f66f6ca70c802e20766914f86137ddb4a707fc88b10e44a01b6d4ff2b3: Status 404 returned error can't find the container with id 4e01b6f66f6ca70c802e20766914f86137ddb4a707fc88b10e44a01b6d4ff2b3
Nov 03 13:34:23.614541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.614509    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"]
Nov 03 13:34:23.869345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.869245    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh" event={"ID":"2f3425a2-8dc5-4666-bf51-b52abeb43601","Type":"ContainerStarted","Data":"b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5"}
Nov 03 13:34:23.869345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.869310    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh" event={"ID":"2f3425a2-8dc5-4666-bf51-b52abeb43601","Type":"ContainerStarted","Data":"4e01b6f66f6ca70c802e20766914f86137ddb4a707fc88b10e44a01b6d4ff2b3"}
Nov 03 13:34:23.869345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.869337    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:23.890136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.890087    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh" podStartSLOduration=1.8900728359999999 podStartE2EDuration="1.890072836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:23.888006711 +0000 UTC m=+1504.965771739" watchObservedRunningTime="2026-11-03 13:34:23.890072836 +0000 UTC m=+1504.967837862"
Nov 03 13:34:23.935509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:23.935480    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:24.218001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:24.217968    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:24.428113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:24.428021    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:24.733786 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:24.733752    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:34:24.948959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:24.942813    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:25.786382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:25.786350    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:26.714415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:26.714358    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:27.418344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:27.414723    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:28.109103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:28.109068    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:29.043643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.043598    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:29.591289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.591254    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"]
Nov 03 13:34:29.591963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.591949    5420 cpu_manager.go:414] "containerMap: removing container" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy"
Nov 03 13:34:29.592007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.591964    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cc5a21-b6ec-49b4-8d77-51fa0a5a1710" containerName="istio-proxy"
Nov 03 13:34:29.595963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.595943    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.602697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.602668    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:34:29.603088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.603067    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:34:29.605086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.605065    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:34:29.605342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.605325    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:34:29.605531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.605507    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"]
Nov 03 13:34:29.630917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.630890    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdb2n\" (UniqueName: \"kubernetes.io/projected/80dc8163-8344-447f-8ebe-1f802a157cf3-kube-api-access-pdb2n\") pod \"maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.631039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.630931    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.631163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.631139    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.732434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.732406    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.732569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.732511    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.732674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.732663    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdb2n\" (UniqueName: \"kubernetes.io/projected/80dc8163-8344-447f-8ebe-1f802a157cf3-kube-api-access-pdb2n\") pod \"maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.733124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.733103    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.734729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.734710    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.741462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.741434    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdb2n\" (UniqueName: \"kubernetes.io/projected/80dc8163-8344-447f-8ebe-1f802a157cf3-kube-api-access-pdb2n\") pod \"maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:29.778739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.778688    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:29.913530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:29.913443    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:30.211705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:30.211675    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"]
Nov 03 13:34:30.212018 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:34:30.211752    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dc8163_8344_447f_8ebe_1f802a157cf3.slice/crio-7e4e3a1ca3f0512ef9e1b796e324230ffe8e82a858e3ad7f801049e02784e490 WatchSource:0}: Error finding container 7e4e3a1ca3f0512ef9e1b796e324230ffe8e82a858e3ad7f801049e02784e490: Status 404 returned error can't find the container with id 7e4e3a1ca3f0512ef9e1b796e324230ffe8e82a858e3ad7f801049e02784e490
Nov 03 13:34:30.536355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:30.536325    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:30.930900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:30.930823    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5" event={"ID":"80dc8163-8344-447f-8ebe-1f802a157cf3","Type":"ContainerStarted","Data":"d1d6e3ce9067e3b5219359ee0ac90b7acab6d9023252419997e93d321cac3da2"}
Nov 03 13:34:30.930900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:30.930873    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5" event={"ID":"80dc8163-8344-447f-8ebe-1f802a157cf3","Type":"ContainerStarted","Data":"7e4e3a1ca3f0512ef9e1b796e324230ffe8e82a858e3ad7f801049e02784e490"}
Nov 03 13:34:30.968232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:30.968160    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5" podStartSLOduration=1.968142074 podStartE2EDuration="1.968142074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:30.961832647 +0000 UTC m=+1512.039597674" watchObservedRunningTime="2026-11-03 13:34:30.968142074 +0000 UTC m=+1512.045907102"
Nov 03 13:34:31.199285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:31.197171    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:31.822482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:31.820243    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:31.941231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:31.941181    5420 generic.go:356] "Generic (PLEG): container finished" podID="80dc8163-8344-447f-8ebe-1f802a157cf3" containerID="d1d6e3ce9067e3b5219359ee0ac90b7acab6d9023252419997e93d321cac3da2" exitCode=0
Nov 03 13:34:31.941387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:31.941238    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5" event={"ID":"80dc8163-8344-447f-8ebe-1f802a157cf3","Type":"ContainerDied","Data":"d1d6e3ce9067e3b5219359ee0ac90b7acab6d9023252419997e93d321cac3da2"}
Nov 03 13:34:32.408972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:32.407587    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:33.123884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.123745    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:33.140155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.139806    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:33.219907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.219883    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dc8163-8344-447f-8ebe-1f802a157cf3-kube-api-access-pdb2n\" (UniqueName: \"kubernetes.io/projected/80dc8163-8344-447f-8ebe-1f802a157cf3-kube-api-access-pdb2n\") pod \"80dc8163-8344-447f-8ebe-1f802a157cf3\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") "
Nov 03 13:34:33.220128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.220107    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-xks-ca\") pod \"80dc8163-8344-447f-8ebe-1f802a157cf3\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") "
Nov 03 13:34:33.220197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.220146    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-openshift-ca\") pod \"80dc8163-8344-447f-8ebe-1f802a157cf3\" (UID: \"80dc8163-8344-447f-8ebe-1f802a157cf3\") "
Nov 03 13:34:33.220598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.220576    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-openshift-ca" pod "80dc8163-8344-447f-8ebe-1f802a157cf3" (UID: "80dc8163-8344-447f-8ebe-1f802a157cf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:33.220941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.220921    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:33.222116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.222088    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dc8163-8344-447f-8ebe-1f802a157cf3-kube-api-access-pdb2n" pod "80dc8163-8344-447f-8ebe-1f802a157cf3" (UID: "80dc8163-8344-447f-8ebe-1f802a157cf3"). InnerVolumeSpecName "kube-api-access-pdb2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:33.222594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.222574    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-xks-ca" pod "80dc8163-8344-447f-8ebe-1f802a157cf3" (UID: "80dc8163-8344-447f-8ebe-1f802a157cf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:34:33.322368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.322329    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80dc8163-8344-447f-8ebe-1f802a157cf3-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:33.322368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.322367    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdb2n\" (UniqueName: \"kubernetes.io/projected/80dc8163-8344-447f-8ebe-1f802a157cf3-kube-api-access-pdb2n\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:33.730975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.730937    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:33.898294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.898261    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:33.960116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.960074    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5" event={"ID":"80dc8163-8344-447f-8ebe-1f802a157cf3","Type":"ContainerDied","Data":"7e4e3a1ca3f0512ef9e1b796e324230ffe8e82a858e3ad7f801049e02784e490"}
Nov 03 13:34:33.960297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.960124    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e4e3a1ca3f0512ef9e1b796e324230ffe8e82a858e3ad7f801049e02784e490"
Nov 03 13:34:33.960297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:33.960084    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"
Nov 03 13:34:34.711264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:34.708181    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:34.879463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:34.879433    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:34:35.575883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:35.575855    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:36.131237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:36.130602    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:36.767224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:36.767182    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:37.533449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:37.533412    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:38.228915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:38.228882    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:38.936634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:38.933828    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:39.620053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:39.620013    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:39.902026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:39.901950    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"]
Nov 03 13:34:39.918107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:39.918081    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"]
Nov 03 13:34:40.041320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:40.041273    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:40.177935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:40.177855    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:40.327174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:40.327130    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:40.821042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:40.821007    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:34:41.216597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:41.216514    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:42.106692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:42.106618    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:42.855669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:42.855632    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:43.768234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:43.767789    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:44.626127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:44.626060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:44.903098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:44.902982    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm" podUID="c7d5ed30-57ee-4d54-8402-7677917247d0" containerName="payload-processing" containerID="cri-o://3d07d7ab64981acd07a68b3dc02b78f405506cf14d65b84de157509719d9df4d" gracePeriod=25
Nov 03 13:34:44.918895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:44.918850    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc" podUID="b079a4df-8022-4fdb-b26d-34791a3523cb" containerName="payload-pre-processing" containerID="cri-o://0041b2dde555d9068b77a7573369ee6f03a1279fece86da7ccf9964e547d2b46" gracePeriod=25
Nov 03 13:34:45.331586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:45.331554    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:46.275011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:46.274968    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:47.064758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:47.064696    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:47.719919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:47.719873    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:47.911967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:47.911923    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:48.150849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:48.150770    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:48.566823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:48.566786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:48.721238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:48.720278    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:49.466403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:49.466353    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:50.165418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:50.165361    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:51.096436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.096402    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:51.584225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.584193    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"]
Nov 03 13:34:51.584927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.584912    5420 cpu_manager.go:414] "containerMap: removing container" podUID="80dc8163-8344-447f-8ebe-1f802a157cf3" containerName="revoke-keys"
Nov 03 13:34:51.584972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.584928    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dc8163-8344-447f-8ebe-1f802a157cf3" containerName="revoke-keys"
Nov 03 13:34:51.585042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.585033    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dc8163-8344-447f-8ebe-1f802a157cf3" containerName="revoke-keys"
Nov 03 13:34:51.588177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.588156    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.592876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.592852    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:34:51.599252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.594448    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:34:51.599252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.594513    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:34:51.599252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.595034    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:34:51.600553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.600531    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"]
Nov 03 13:34:51.737796 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.737766    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3412a92-5rn7x\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.737958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.737801    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zx7\" (UniqueName: \"kubernetes.io/projected/b113f1ba-d494-4300-a9ee-f979695a60c5-kube-api-access-26zx7\") pod \"maas-api-revoke-sub-d3412a92-5rn7x\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.737958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.737950    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3412a92-5rn7x\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.839580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.839486    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3412a92-5rn7x\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.839580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.839520    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zx7\" (UniqueName: \"kubernetes.io/projected/b113f1ba-d494-4300-a9ee-f979695a60c5-kube-api-access-26zx7\") pod \"maas-api-revoke-sub-d3412a92-5rn7x\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.839830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.839688    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3412a92-5rn7x\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.840386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.840362    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3412a92-5rn7x\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.842042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.842023    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3412a92-5rn7x\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.848683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.848655    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zx7\" (UniqueName: \"kubernetes.io/projected/b113f1ba-d494-4300-a9ee-f979695a60c5-kube-api-access-26zx7\") pod \"maas-api-revoke-sub-d3412a92-5rn7x\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.905106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.905081    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:51.976146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.976115    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"]
Nov 03 13:34:51.984112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.983891    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:51.989926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:51.989892    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"]
Nov 03 13:34:52.043505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.043468    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ba37af-q86v2\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.043657 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.043608    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4psk\" (UniqueName: \"kubernetes.io/projected/8b1e7472-ef8a-43b3-b515-778af59b79f4-kube-api-access-z4psk\") pod \"maas-api-revoke-sub-e2ba37af-q86v2\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.043746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.043726    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ba37af-q86v2\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.145366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.145278    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ba37af-q86v2\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.145366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.145344    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4psk\" (UniqueName: \"kubernetes.io/projected/8b1e7472-ef8a-43b3-b515-778af59b79f4-kube-api-access-z4psk\") pod \"maas-api-revoke-sub-e2ba37af-q86v2\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.145584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.145409    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ba37af-q86v2\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.146023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.146005    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ba37af-q86v2\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.147627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.147605    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ba37af-q86v2\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.154016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.153996    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4psk\" (UniqueName: \"kubernetes.io/projected/8b1e7472-ef8a-43b3-b515-778af59b79f4-kube-api-access-z4psk\") pod \"maas-api-revoke-sub-e2ba37af-q86v2\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.187755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.187731    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"]
Nov 03 13:34:52.187862 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:34:52.187806    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb113f1ba_d494_4300_a9ee_f979695a60c5.slice/crio-2af14cf88c627980eb14e678775adc20209c39f86c842b6e1d12180b18118fc4 WatchSource:0}: Error finding container 2af14cf88c627980eb14e678775adc20209c39f86c842b6e1d12180b18118fc4: Status 404 returned error can't find the container with id 2af14cf88c627980eb14e678775adc20209c39f86c842b6e1d12180b18118fc4
Nov 03 13:34:52.323820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.323792    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:52.384308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.383245    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:52.483946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.483911    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"]
Nov 03 13:34:52.490197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.490168    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.498498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.498471    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"]
Nov 03 13:34:52.552513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.552485    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a29ce8b3-lmtt7\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.552800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.552778    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q4fz\" (UniqueName: \"kubernetes.io/projected/6fc96b77-2e71-4e30-8ef3-9bec7976814a-kube-api-access-7q4fz\") pod \"maas-api-revoke-sub-a29ce8b3-lmtt7\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.552867 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.552815    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a29ce8b3-lmtt7\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.563692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.563669    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:52.654685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.654663    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4fz\" (UniqueName: \"kubernetes.io/projected/6fc96b77-2e71-4e30-8ef3-9bec7976814a-kube-api-access-7q4fz\") pod \"maas-api-revoke-sub-a29ce8b3-lmtt7\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.655006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.654701    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a29ce8b3-lmtt7\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.655006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.654788    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a29ce8b3-lmtt7\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.655600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.655576    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a29ce8b3-lmtt7\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.656720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.656693    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"]
Nov 03 13:34:52.656947 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:34:52.656927    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1e7472_ef8a_43b3_b515_778af59b79f4.slice/crio-8f5f50240bfa0d8ba26c0aeb905d0343ed3a5f4b2189fad264baf68b56ebfeed WatchSource:0}: Error finding container 8f5f50240bfa0d8ba26c0aeb905d0343ed3a5f4b2189fad264baf68b56ebfeed: Status 404 returned error can't find the container with id 8f5f50240bfa0d8ba26c0aeb905d0343ed3a5f4b2189fad264baf68b56ebfeed
Nov 03 13:34:52.657035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.656985    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a29ce8b3-lmtt7\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.663432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.663411    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4fz\" (UniqueName: \"kubernetes.io/projected/6fc96b77-2e71-4e30-8ef3-9bec7976814a-kube-api-access-7q4fz\") pod \"maas-api-revoke-sub-a29ce8b3-lmtt7\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:52.813574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:52.813546    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:53.099659 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:34:53.099635    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc96b77_2e71_4e30_8ef3_9bec7976814a.slice/crio-3105b72a3b5da7360be86dcf09e5e62a5f7f4e79d9331602df850123ae52c11a WatchSource:0}: Error finding container 3105b72a3b5da7360be86dcf09e5e62a5f7f4e79d9331602df850123ae52c11a: Status 404 returned error can't find the container with id 3105b72a3b5da7360be86dcf09e5e62a5f7f4e79d9331602df850123ae52c11a
Nov 03 13:34:53.099977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.099954    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"]
Nov 03 13:34:53.184370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.184341    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x" event={"ID":"b113f1ba-d494-4300-a9ee-f979695a60c5","Type":"ContainerStarted","Data":"8fb5212fb05732ad42ce22b6e464d3f542b1f64f19868dd27f57e8276042a91d"}
Nov 03 13:34:53.184760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.184401    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x" event={"ID":"b113f1ba-d494-4300-a9ee-f979695a60c5","Type":"ContainerStarted","Data":"2af14cf88c627980eb14e678775adc20209c39f86c842b6e1d12180b18118fc4"}
Nov 03 13:34:53.186052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.186026    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2" event={"ID":"8b1e7472-ef8a-43b3-b515-778af59b79f4","Type":"ContainerStarted","Data":"069e2b3db24ccf688410195e65a328f856c5d625b6a5a2dc1420e871c1c6d235"}
Nov 03 13:34:53.186143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.186063    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2" event={"ID":"8b1e7472-ef8a-43b3-b515-778af59b79f4","Type":"ContainerStarted","Data":"8f5f50240bfa0d8ba26c0aeb905d0343ed3a5f4b2189fad264baf68b56ebfeed"}
Nov 03 13:34:53.187386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.187367    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7" event={"ID":"6fc96b77-2e71-4e30-8ef3-9bec7976814a","Type":"ContainerStarted","Data":"3105b72a3b5da7360be86dcf09e5e62a5f7f4e79d9331602df850123ae52c11a"}
Nov 03 13:34:53.194471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.194448    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc" podUID="b079a4df-8022-4fdb-b26d-34791a3523cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:34:53.194574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.194476    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm" podUID="c7d5ed30-57ee-4d54-8402-7677917247d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:34:53.206474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.206438    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x" podStartSLOduration=2.206426825 podStartE2EDuration="2.206426825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:53.203651158 +0000 UTC m=+1534.281416167" watchObservedRunningTime="2026-11-03 13:34:53.206426825 +0000 UTC m=+1534.284191885"
Nov 03 13:34:53.223846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.223805    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2" podStartSLOduration=2.223792943 podStartE2EDuration="2.223792943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:53.222687573 +0000 UTC m=+1534.300452600" watchObservedRunningTime="2026-11-03 13:34:53.223792943 +0000 UTC m=+1534.301557968"
Nov 03 13:34:53.447228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.447187    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:53.778954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.778920    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:53.987234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:53.980850    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:54.196843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:54.196803    5420 generic.go:356] "Generic (PLEG): container finished" podID="6fc96b77-2e71-4e30-8ef3-9bec7976814a" containerID="4ac8bfadd66a80809f03fc3da4603a3037e0e5bf3630792f6c8093e7a811653e" exitCode=0
Nov 03 13:34:54.196995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:54.196891    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7" event={"ID":"6fc96b77-2e71-4e30-8ef3-9bec7976814a","Type":"ContainerDied","Data":"4ac8bfadd66a80809f03fc3da4603a3037e0e5bf3630792f6c8093e7a811653e"}
Nov 03 13:34:54.198897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:54.198870    5420 generic.go:356] "Generic (PLEG): container finished" podID="b113f1ba-d494-4300-a9ee-f979695a60c5" containerID="8fb5212fb05732ad42ce22b6e464d3f542b1f64f19868dd27f57e8276042a91d" exitCode=0
Nov 03 13:34:54.199073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:54.198950    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x" event={"ID":"b113f1ba-d494-4300-a9ee-f979695a60c5","Type":"ContainerDied","Data":"8fb5212fb05732ad42ce22b6e464d3f542b1f64f19868dd27f57e8276042a91d"}
Nov 03 13:34:54.200902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:54.200874    5420 generic.go:356] "Generic (PLEG): container finished" podID="8b1e7472-ef8a-43b3-b515-778af59b79f4" containerID="069e2b3db24ccf688410195e65a328f856c5d625b6a5a2dc1420e871c1c6d235" exitCode=0
Nov 03 13:34:54.201036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:54.200951    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2" event={"ID":"8b1e7472-ef8a-43b3-b515-778af59b79f4","Type":"ContainerDied","Data":"069e2b3db24ccf688410195e65a328f856c5d625b6a5a2dc1420e871c1c6d235"}
Nov 03 13:34:54.422436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:54.422251    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:54.733104 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:34:54.733067    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:34:55.122036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.121720    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:55.411664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.411638    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:55.468689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.468660    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:55.470714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.470691    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:55.538769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.538734    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc96b77-2e71-4e30-8ef3-9bec7976814a-kube-api-access-7q4fz\" (UniqueName: \"kubernetes.io/projected/6fc96b77-2e71-4e30-8ef3-9bec7976814a-kube-api-access-7q4fz\") pod \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") "
Nov 03 13:34:55.538935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.538840    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-openshift-ca\") pod \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") "
Nov 03 13:34:55.539006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.538989    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-xks-ca\") pod \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\" (UID: \"6fc96b77-2e71-4e30-8ef3-9bec7976814a\") "
Nov 03 13:34:55.539294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.539263    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-openshift-ca" pod "6fc96b77-2e71-4e30-8ef3-9bec7976814a" (UID: "6fc96b77-2e71-4e30-8ef3-9bec7976814a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:55.539772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.539745    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:55.541039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.541013    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc96b77-2e71-4e30-8ef3-9bec7976814a-kube-api-access-7q4fz" pod "6fc96b77-2e71-4e30-8ef3-9bec7976814a" (UID: "6fc96b77-2e71-4e30-8ef3-9bec7976814a"). InnerVolumeSpecName "kube-api-access-7q4fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:55.541133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.541062    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-xks-ca" pod "6fc96b77-2e71-4e30-8ef3-9bec7976814a" (UID: "6fc96b77-2e71-4e30-8ef3-9bec7976814a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:34:55.640648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.640585    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-openshift-ca\") pod \"b113f1ba-d494-4300-a9ee-f979695a60c5\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") "
Nov 03 13:34:55.640781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.640656    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b113f1ba-d494-4300-a9ee-f979695a60c5-kube-api-access-26zx7\" (UniqueName: \"kubernetes.io/projected/b113f1ba-d494-4300-a9ee-f979695a60c5-kube-api-access-26zx7\") pod \"b113f1ba-d494-4300-a9ee-f979695a60c5\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") "
Nov 03 13:34:55.640781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.640735    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1e7472-ef8a-43b3-b515-778af59b79f4-kube-api-access-z4psk\" (UniqueName: \"kubernetes.io/projected/8b1e7472-ef8a-43b3-b515-778af59b79f4-kube-api-access-z4psk\") pod \"8b1e7472-ef8a-43b3-b515-778af59b79f4\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") "
Nov 03 13:34:55.640781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.640777    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-xks-ca\") pod \"8b1e7472-ef8a-43b3-b515-778af59b79f4\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") "
Nov 03 13:34:55.640975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.640948    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-openshift-ca" pod "b113f1ba-d494-4300-a9ee-f979695a60c5" (UID: "b113f1ba-d494-4300-a9ee-f979695a60c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:55.641039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.640988    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-xks-ca\") pod \"b113f1ba-d494-4300-a9ee-f979695a60c5\" (UID: \"b113f1ba-d494-4300-a9ee-f979695a60c5\") "
Nov 03 13:34:55.641039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.641029    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-openshift-ca\") pod \"8b1e7472-ef8a-43b3-b515-778af59b79f4\" (UID: \"8b1e7472-ef8a-43b3-b515-778af59b79f4\") "
Nov 03 13:34:55.641477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.641452    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-openshift-ca" pod "8b1e7472-ef8a-43b3-b515-778af59b79f4" (UID: "8b1e7472-ef8a-43b3-b515-778af59b79f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:34:55.641888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.641863    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc96b77-2e71-4e30-8ef3-9bec7976814a-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:55.641998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.641895    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:55.641998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.641913    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:55.641998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.641928    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q4fz\" (UniqueName: \"kubernetes.io/projected/6fc96b77-2e71-4e30-8ef3-9bec7976814a-kube-api-access-7q4fz\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:55.642961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.642939    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-xks-ca" pod "8b1e7472-ef8a-43b3-b515-778af59b79f4" (UID: "8b1e7472-ef8a-43b3-b515-778af59b79f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:34:55.643050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.643020    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b113f1ba-d494-4300-a9ee-f979695a60c5-kube-api-access-26zx7" pod "b113f1ba-d494-4300-a9ee-f979695a60c5" (UID: "b113f1ba-d494-4300-a9ee-f979695a60c5"). InnerVolumeSpecName "kube-api-access-26zx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:55.643050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.643037    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-xks-ca" pod "b113f1ba-d494-4300-a9ee-f979695a60c5" (UID: "b113f1ba-d494-4300-a9ee-f979695a60c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:34:55.643150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.643049    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1e7472-ef8a-43b3-b515-778af59b79f4-kube-api-access-z4psk" pod "8b1e7472-ef8a-43b3-b515-778af59b79f4" (UID: "8b1e7472-ef8a-43b3-b515-778af59b79f4"). InnerVolumeSpecName "kube-api-access-z4psk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:34:55.743818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.743781    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113f1ba-d494-4300-a9ee-f979695a60c5-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:55.743818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.743819    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zx7\" (UniqueName: \"kubernetes.io/projected/b113f1ba-d494-4300-a9ee-f979695a60c5-kube-api-access-26zx7\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:55.744059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.743844    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4psk\" (UniqueName: \"kubernetes.io/projected/8b1e7472-ef8a-43b3-b515-778af59b79f4-kube-api-access-z4psk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:55.744059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.743865    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1e7472-ef8a-43b3-b515-778af59b79f4-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:34:55.806948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:55.805640    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:56.218279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.218247    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2" event={"ID":"8b1e7472-ef8a-43b3-b515-778af59b79f4","Type":"ContainerDied","Data":"8f5f50240bfa0d8ba26c0aeb905d0343ed3a5f4b2189fad264baf68b56ebfeed"}
Nov 03 13:34:56.218693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.218290    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"
Nov 03 13:34:56.218693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.218297    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f5f50240bfa0d8ba26c0aeb905d0343ed3a5f4b2189fad264baf68b56ebfeed"
Nov 03 13:34:56.219951 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.219931    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"
Nov 03 13:34:56.220090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.219968    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7" event={"ID":"6fc96b77-2e71-4e30-8ef3-9bec7976814a","Type":"ContainerDied","Data":"3105b72a3b5da7360be86dcf09e5e62a5f7f4e79d9331602df850123ae52c11a"}
Nov 03 13:34:56.220090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.220009    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3105b72a3b5da7360be86dcf09e5e62a5f7f4e79d9331602df850123ae52c11a"
Nov 03 13:34:56.221446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.221423    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x" event={"ID":"b113f1ba-d494-4300-a9ee-f979695a60c5","Type":"ContainerDied","Data":"2af14cf88c627980eb14e678775adc20209c39f86c842b6e1d12180b18118fc4"}
Nov 03 13:34:56.221446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.221459    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2af14cf88c627980eb14e678775adc20209c39f86c842b6e1d12180b18118fc4"
Nov 03 13:34:56.221626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.221461    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"
Nov 03 13:34:56.437886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:56.437855    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:57.132231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:57.130125    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:57.815823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:57.815739    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:58.589972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:58.589943    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:59.264825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:59.264796    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:59.856890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:34:59.854320    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:00.649091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:00.649013    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:01.318803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:01.317325    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:02.109897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:02.108758    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:02.627913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:02.627882    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:02.797337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:02.797305    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:03.179205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.179177    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:03.203007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.202970    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc" podUID="b079a4df-8022-4fdb-b26d-34791a3523cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:35:03.203611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.202986    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm" podUID="c7d5ed30-57ee-4d54-8402-7677917247d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:35:03.531389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.531353    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"]
Nov 03 13:35:03.532444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.532421    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b113f1ba-d494-4300-a9ee-f979695a60c5" containerName="revoke-keys"
Nov 03 13:35:03.532573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.532445    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b113f1ba-d494-4300-a9ee-f979695a60c5" containerName="revoke-keys"
Nov 03 13:35:03.532573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.532468    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6fc96b77-2e71-4e30-8ef3-9bec7976814a" containerName="revoke-keys"
Nov 03 13:35:03.532573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.532476    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc96b77-2e71-4e30-8ef3-9bec7976814a" containerName="revoke-keys"
Nov 03 13:35:03.532573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.532501    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8b1e7472-ef8a-43b3-b515-778af59b79f4" containerName="revoke-keys"
Nov 03 13:35:03.532573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.532508    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1e7472-ef8a-43b3-b515-778af59b79f4" containerName="revoke-keys"
Nov 03 13:35:03.532848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.532694    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1e7472-ef8a-43b3-b515-778af59b79f4" containerName="revoke-keys"
Nov 03 13:35:03.532848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.532722    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b113f1ba-d494-4300-a9ee-f979695a60c5" containerName="revoke-keys"
Nov 03 13:35:03.532848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.532734    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc96b77-2e71-4e30-8ef3-9bec7976814a" containerName="revoke-keys"
Nov 03 13:35:03.538364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.538339    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.546046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.546019    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:35:03.549832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.549811    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:35:03.550351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.550324    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:35:03.551521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.551500    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:35:03.558732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.558708    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"]
Nov 03 13:35:03.691773 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.691739    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ae821f-b2kwr\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.691990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.691970    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lkqv\" (UniqueName: \"kubernetes.io/projected/b2277d95-deaa-457e-aa15-5351aa31d91d-kube-api-access-9lkqv\") pod \"maas-api-revoke-sub-c7ae821f-b2kwr\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.692057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.692010    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ae821f-b2kwr\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.793737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.793653    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lkqv\" (UniqueName: \"kubernetes.io/projected/b2277d95-deaa-457e-aa15-5351aa31d91d-kube-api-access-9lkqv\") pod \"maas-api-revoke-sub-c7ae821f-b2kwr\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.793737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.793704    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ae821f-b2kwr\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.793952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.793862    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ae821f-b2kwr\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.794396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.794371    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ae821f-b2kwr\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.796280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.796241    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ae821f-b2kwr\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.802080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.802058    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lkqv\" (UniqueName: \"kubernetes.io/projected/b2277d95-deaa-457e-aa15-5351aa31d91d-kube-api-access-9lkqv\") pod \"maas-api-revoke-sub-c7ae821f-b2kwr\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:03.855765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:03.855741    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:04.008993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.008933    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:04.162166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.162025    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:04.205260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.205091    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"]
Nov 03 13:35:04.206379 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:04.206350    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2277d95_deaa_457e_aa15_5351aa31d91d.slice/crio-6da0b4f1b07bee1b33f09dfe8939422f332e9e77053960e45e567e558d53412f WatchSource:0}: Error finding container 6da0b4f1b07bee1b33f09dfe8939422f332e9e77053960e45e567e558d53412f: Status 404 returned error can't find the container with id 6da0b4f1b07bee1b33f09dfe8939422f332e9e77053960e45e567e558d53412f
Nov 03 13:35:04.303765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.303734    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr" event={"ID":"b2277d95-deaa-457e-aa15-5351aa31d91d","Type":"ContainerStarted","Data":"6da0b4f1b07bee1b33f09dfe8939422f332e9e77053960e45e567e558d53412f"}
Nov 03 13:35:04.593100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.593066    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"]
Nov 03 13:35:04.598970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.598948    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.607925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.607897    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"]
Nov 03 13:35:04.732255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.731562    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b573dc-nzdd7\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.732255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.731653    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zml\" (UniqueName: \"kubernetes.io/projected/0ce7938f-84e4-4a81-b519-b100584149a3-kube-api-access-f9zml\") pod \"maas-api-revoke-sub-d7b573dc-nzdd7\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.732255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.731705    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b573dc-nzdd7\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.840866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.840307    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b573dc-nzdd7\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.840866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.840638    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b573dc-nzdd7\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.840866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.840747    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zml\" (UniqueName: \"kubernetes.io/projected/0ce7938f-84e4-4a81-b519-b100584149a3-kube-api-access-f9zml\") pod \"maas-api-revoke-sub-d7b573dc-nzdd7\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.841887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.841833    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b573dc-nzdd7\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.852273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.850651    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b573dc-nzdd7\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.855462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.855433    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zml\" (UniqueName: \"kubernetes.io/projected/0ce7938f-84e4-4a81-b519-b100584149a3-kube-api-access-f9zml\") pod \"maas-api-revoke-sub-d7b573dc-nzdd7\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:04.917364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:04.917184    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:05.009711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.009677    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:05.058247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.057839    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"]
Nov 03 13:35:05.093780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.082363    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"]
Nov 03 13:35:05.093780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.082493    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.196786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.195274    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:05.242985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.242554    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"]
Nov 03 13:35:05.244826 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:05.244796    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce7938f_84e4_4a81_b519_b100584149a3.slice/crio-d32ff102f0901e0e46e7c814ad7b446220456caa995f3590ec14e7217b99fbec WatchSource:0}: Error finding container d32ff102f0901e0e46e7c814ad7b446220456caa995f3590ec14e7217b99fbec: Status 404 returned error can't find the container with id d32ff102f0901e0e46e7c814ad7b446220456caa995f3590ec14e7217b99fbec
Nov 03 13:35:05.258293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.257542    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"]
Nov 03 13:35:05.258744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.258706    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.258902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.258889    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9r6\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-kube-api-access-9g9r6\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.259227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.259177    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.259622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.259601    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.263202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.263179    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1ec4b892-m8gr5"]
Nov 03 13:35:05.330883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.330826    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7" event={"ID":"0ce7938f-84e4-4a81-b519-b100584149a3","Type":"ContainerStarted","Data":"d32ff102f0901e0e46e7c814ad7b446220456caa995f3590ec14e7217b99fbec"}
Nov 03 13:35:05.332854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.332817    5420 generic.go:356] "Generic (PLEG): container finished" podID="c7d5ed30-57ee-4d54-8402-7677917247d0" containerID="3d07d7ab64981acd07a68b3dc02b78f405506cf14d65b84de157509719d9df4d" exitCode=0
Nov 03 13:35:05.333021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.332930    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm" event={"ID":"c7d5ed30-57ee-4d54-8402-7677917247d0","Type":"ContainerDied","Data":"3d07d7ab64981acd07a68b3dc02b78f405506cf14d65b84de157509719d9df4d"}
Nov 03 13:35:05.334923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.334895    5420 generic.go:356] "Generic (PLEG): container finished" podID="b079a4df-8022-4fdb-b26d-34791a3523cb" containerID="0041b2dde555d9068b77a7573369ee6f03a1279fece86da7ccf9964e547d2b46" exitCode=0
Nov 03 13:35:05.335054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.335003    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc" event={"ID":"b079a4df-8022-4fdb-b26d-34791a3523cb","Type":"ContainerDied","Data":"0041b2dde555d9068b77a7573369ee6f03a1279fece86da7ccf9964e547d2b46"}
Nov 03 13:35:05.338118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.337231    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr" event={"ID":"b2277d95-deaa-457e-aa15-5351aa31d91d","Type":"ContainerStarted","Data":"3143d6200e37d6544cabdf2fc5afbe2500cc64d1980920be93f32e700fefd76f"}
Nov 03 13:35:05.361157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.361048    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr" podStartSLOduration=2.3610309640000002 podStartE2EDuration="2.361030964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:05.357623315 +0000 UTC m=+1546.435388335" watchObservedRunningTime="2026-11-03 13:35:05.361030964 +0000 UTC m=+1546.438795991"
Nov 03 13:35:05.361892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.361861    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9r6\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-kube-api-access-9g9r6\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.362012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.361955    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.362152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.362017    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.364713 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.364292    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.365722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.364652    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.366794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.365300    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.368567 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.368527    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:05.446109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.446063    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9r6\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-kube-api-access-9g9r6\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.452147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.452115    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.460273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.460244    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:35:05.465410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.465381    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:35:05.467156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.467117    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b079a4df-8022-4fdb-b26d-34791a3523cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b079a4df-8022-4fdb-b26d-34791a3523cb-plugins-config-volume\") pod \"b079a4df-8022-4fdb-b26d-34791a3523cb\" (UID: \"b079a4df-8022-4fdb-b26d-34791a3523cb\") "
Nov 03 13:35:05.467455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.467246    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b079a4df-8022-4fdb-b26d-34791a3523cb-kube-api-access-p2rng\" (UniqueName: \"kubernetes.io/projected/b079a4df-8022-4fdb-b26d-34791a3523cb-kube-api-access-p2rng\") pod \"b079a4df-8022-4fdb-b26d-34791a3523cb\" (UID: \"b079a4df-8022-4fdb-b26d-34791a3523cb\") "
Nov 03 13:35:05.467555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.467521    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7d5ed30-57ee-4d54-8402-7677917247d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d5ed30-57ee-4d54-8402-7677917247d0-plugins-config-volume\") pod \"c7d5ed30-57ee-4d54-8402-7677917247d0\" (UID: \"c7d5ed30-57ee-4d54-8402-7677917247d0\") "
Nov 03 13:35:05.467648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.467588    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d5ed30-57ee-4d54-8402-7677917247d0-kube-api-access-75fvs\" (UniqueName: \"kubernetes.io/projected/c7d5ed30-57ee-4d54-8402-7677917247d0-kube-api-access-75fvs\") pod \"c7d5ed30-57ee-4d54-8402-7677917247d0\" (UID: \"c7d5ed30-57ee-4d54-8402-7677917247d0\") "
Nov 03 13:35:05.468125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.468098    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d5ed30-57ee-4d54-8402-7677917247d0-plugins-config-volume" pod "c7d5ed30-57ee-4d54-8402-7677917247d0" (UID: "c7d5ed30-57ee-4d54-8402-7677917247d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:05.468562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.468528    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b079a4df-8022-4fdb-b26d-34791a3523cb-plugins-config-volume" pod "b079a4df-8022-4fdb-b26d-34791a3523cb" (UID: "b079a4df-8022-4fdb-b26d-34791a3523cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:05.470924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.470891    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d5ed30-57ee-4d54-8402-7677917247d0-kube-api-access-75fvs" pod "c7d5ed30-57ee-4d54-8402-7677917247d0" (UID: "c7d5ed30-57ee-4d54-8402-7677917247d0"). InnerVolumeSpecName "kube-api-access-75fvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:05.471497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.471470    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b079a4df-8022-4fdb-b26d-34791a3523cb-kube-api-access-p2rng" pod "b079a4df-8022-4fdb-b26d-34791a3523cb" (UID: "b079a4df-8022-4fdb-b26d-34791a3523cb"). InnerVolumeSpecName "kube-api-access-p2rng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:05.474262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.474239    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:05.571153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.571089    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2rng\" (UniqueName: \"kubernetes.io/projected/b079a4df-8022-4fdb-b26d-34791a3523cb-kube-api-access-p2rng\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:05.571153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.571117    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d5ed30-57ee-4d54-8402-7677917247d0-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:05.571153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.571126    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75fvs\" (UniqueName: \"kubernetes.io/projected/c7d5ed30-57ee-4d54-8402-7677917247d0-kube-api-access-75fvs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:05.571153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.571136    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b079a4df-8022-4fdb-b26d-34791a3523cb-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:05.847155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:05.847128    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"]
Nov 03 13:35:05.847726 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:05.846200    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69480ba8_e96f_447d_af1e_2d78ab7116c5.slice/crio-1f0c20bc4c4ddd724771e9eadf73cc5d88caeed74d862d9ae1dfbc72655ef271 WatchSource:0}: Error finding container 1f0c20bc4c4ddd724771e9eadf73cc5d88caeed74d862d9ae1dfbc72655ef271: Status 404 returned error can't find the container with id 1f0c20bc4c4ddd724771e9eadf73cc5d88caeed74d862d9ae1dfbc72655ef271
Nov 03 13:35:06.349368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.349343    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"
Nov 03 13:35:06.349777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.349376    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm" event={"ID":"c7d5ed30-57ee-4d54-8402-7677917247d0","Type":"ContainerDied","Data":"058f17bff346c6346acd46c8219ca78c996438b9593ac0cae037e48dfbc5654d"}
Nov 03 13:35:06.349777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.349436    5420 scope.go:122] "RemoveContainer" containerID="3d07d7ab64981acd07a68b3dc02b78f405506cf14d65b84de157509719d9df4d"
Nov 03 13:35:06.351748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.351718    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs" event={"ID":"69480ba8-e96f-447d-af1e-2d78ab7116c5","Type":"ContainerStarted","Data":"fa9d8f801b8bf27e5725d876ea5fca5e4a13df7039788379b76bd2787567253e"}
Nov 03 13:35:06.351870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.351810    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:06.351870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.351836    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs" event={"ID":"69480ba8-e96f-447d-af1e-2d78ab7116c5","Type":"ContainerStarted","Data":"1f0c20bc4c4ddd724771e9eadf73cc5d88caeed74d862d9ae1dfbc72655ef271"}
Nov 03 13:35:06.358199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.357060    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"
Nov 03 13:35:06.358199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.357089    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc" event={"ID":"b079a4df-8022-4fdb-b26d-34791a3523cb","Type":"ContainerDied","Data":"524c930a251927f3ec03010f53a0315dbf99632b945e7f32323091cccfbe9b2e"}
Nov 03 13:35:06.359663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.359639    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:06.360405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.360379    5420 generic.go:356] "Generic (PLEG): container finished" podID="b2277d95-deaa-457e-aa15-5351aa31d91d" containerID="3143d6200e37d6544cabdf2fc5afbe2500cc64d1980920be93f32e700fefd76f" exitCode=0
Nov 03 13:35:06.360512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.360449    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr" event={"ID":"b2277d95-deaa-457e-aa15-5351aa31d91d","Type":"ContainerDied","Data":"3143d6200e37d6544cabdf2fc5afbe2500cc64d1980920be93f32e700fefd76f"}
Nov 03 13:35:06.362090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.362069    5420 generic.go:356] "Generic (PLEG): container finished" podID="0ce7938f-84e4-4a81-b519-b100584149a3" containerID="84b36ea7cd890535538783c620abdc84682537fee2bec309072e6f25cef62513" exitCode=0
Nov 03 13:35:06.362174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.362144    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7" event={"ID":"0ce7938f-84e4-4a81-b519-b100584149a3","Type":"ContainerDied","Data":"84b36ea7cd890535538783c620abdc84682537fee2bec309072e6f25cef62513"}
Nov 03 13:35:06.384590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.384526    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs" podStartSLOduration=1.3845021659999999 podStartE2EDuration="1.384502166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:06.378421681 +0000 UTC m=+1547.456186734" watchObservedRunningTime="2026-11-03 13:35:06.384502166 +0000 UTC m=+1547.462267193"
Nov 03 13:35:06.424444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.416461    5420 scope.go:122] "RemoveContainer" containerID="0041b2dde555d9068b77a7573369ee6f03a1279fece86da7ccf9964e547d2b46"
Nov 03 13:35:06.479106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.479072    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"]
Nov 03 13:35:06.487372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.487343    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-2qhdm"]
Nov 03 13:35:06.514593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.514563    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"]
Nov 03 13:35:06.526065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:06.526038    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-d5bzc"]
Nov 03 13:35:07.086336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.086307    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80dc8163-8344-447f-8ebe-1f802a157cf3" path="/var/lib/kubelet/pods/80dc8163-8344-447f-8ebe-1f802a157cf3/volumes"
Nov 03 13:35:07.086737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.086723    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b079a4df-8022-4fdb-b26d-34791a3523cb" path="/var/lib/kubelet/pods/b079a4df-8022-4fdb-b26d-34791a3523cb/volumes"
Nov 03 13:35:07.087095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.087084    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7d5ed30-57ee-4d54-8402-7677917247d0" path="/var/lib/kubelet/pods/c7d5ed30-57ee-4d54-8402-7677917247d0/volumes"
Nov 03 13:35:07.504375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.504124    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:07.597049 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.597026    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:07.614983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.614949    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2277d95-deaa-457e-aa15-5351aa31d91d-kube-api-access-9lkqv\" (UniqueName: \"kubernetes.io/projected/b2277d95-deaa-457e-aa15-5351aa31d91d-kube-api-access-9lkqv\") pod \"b2277d95-deaa-457e-aa15-5351aa31d91d\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") "
Nov 03 13:35:07.615107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.615033    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-openshift-ca\") pod \"b2277d95-deaa-457e-aa15-5351aa31d91d\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") "
Nov 03 13:35:07.615395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.615370    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-xks-ca\") pod \"b2277d95-deaa-457e-aa15-5351aa31d91d\" (UID: \"b2277d95-deaa-457e-aa15-5351aa31d91d\") "
Nov 03 13:35:07.615520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.615394    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-openshift-ca" pod "b2277d95-deaa-457e-aa15-5351aa31d91d" (UID: "b2277d95-deaa-457e-aa15-5351aa31d91d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:07.616019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.615993    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:07.617291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.617201    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2277d95-deaa-457e-aa15-5351aa31d91d-kube-api-access-9lkqv" pod "b2277d95-deaa-457e-aa15-5351aa31d91d" (UID: "b2277d95-deaa-457e-aa15-5351aa31d91d"). InnerVolumeSpecName "kube-api-access-9lkqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:07.617410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.617351    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-xks-ca" pod "b2277d95-deaa-457e-aa15-5351aa31d91d" (UID: "b2277d95-deaa-457e-aa15-5351aa31d91d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:07.654546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.654521    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:07.717172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.717140    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce7938f-84e4-4a81-b519-b100584149a3-kube-api-access-f9zml\" (UniqueName: \"kubernetes.io/projected/0ce7938f-84e4-4a81-b519-b100584149a3-kube-api-access-f9zml\") pod \"0ce7938f-84e4-4a81-b519-b100584149a3\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") "
Nov 03 13:35:07.717377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.717265    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-xks-ca\") pod \"0ce7938f-84e4-4a81-b519-b100584149a3\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") "
Nov 03 13:35:07.717377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.717370    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-openshift-ca\") pod \"0ce7938f-84e4-4a81-b519-b100584149a3\" (UID: \"0ce7938f-84e4-4a81-b519-b100584149a3\") "
Nov 03 13:35:07.717732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.717705    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-openshift-ca" pod "0ce7938f-84e4-4a81-b519-b100584149a3" (UID: "0ce7938f-84e4-4a81-b519-b100584149a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:07.718251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.718191    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lkqv\" (UniqueName: \"kubernetes.io/projected/b2277d95-deaa-457e-aa15-5351aa31d91d-kube-api-access-9lkqv\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:07.718314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.718259    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:07.718314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.718277    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2277d95-deaa-457e-aa15-5351aa31d91d-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:07.719521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.719497    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce7938f-84e4-4a81-b519-b100584149a3-kube-api-access-f9zml" pod "0ce7938f-84e4-4a81-b519-b100584149a3" (UID: "0ce7938f-84e4-4a81-b519-b100584149a3"). InnerVolumeSpecName "kube-api-access-f9zml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:07.719621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.719497    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-xks-ca" pod "0ce7938f-84e4-4a81-b519-b100584149a3" (UID: "0ce7938f-84e4-4a81-b519-b100584149a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:07.820075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.820040    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9zml\" (UniqueName: \"kubernetes.io/projected/0ce7938f-84e4-4a81-b519-b100584149a3-kube-api-access-f9zml\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:07.820075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:07.820072    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce7938f-84e4-4a81-b519-b100584149a3-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:08.244929 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:08.244896    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:08.382027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:08.382000    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"
Nov 03 13:35:08.382238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:08.382035    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7" event={"ID":"0ce7938f-84e4-4a81-b519-b100584149a3","Type":"ContainerDied","Data":"d32ff102f0901e0e46e7c814ad7b446220456caa995f3590ec14e7217b99fbec"}
Nov 03 13:35:08.382238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:08.382083    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d32ff102f0901e0e46e7c814ad7b446220456caa995f3590ec14e7217b99fbec"
Nov 03 13:35:08.383588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:08.383568    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"
Nov 03 13:35:08.383588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:08.383576    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr" event={"ID":"b2277d95-deaa-457e-aa15-5351aa31d91d","Type":"ContainerDied","Data":"6da0b4f1b07bee1b33f09dfe8939422f332e9e77053960e45e567e558d53412f"}
Nov 03 13:35:08.383789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:08.383610    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da0b4f1b07bee1b33f09dfe8939422f332e9e77053960e45e567e558d53412f"
Nov 03 13:35:08.956392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:08.955767    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:09.557162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:09.557134    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:09.719060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:09.719018    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:10.404390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:10.404358    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:11.139236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:11.137089    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:11.833235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:11.832590    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:12.616864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:12.616830    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:13.004411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.004376    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:13.144551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.144519    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:13.404625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.404527    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:13.484813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.484780    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"]
Nov 03 13:35:13.485471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485457    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c7d5ed30-57ee-4d54-8402-7677917247d0" containerName="payload-processing"
Nov 03 13:35:13.485537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485472    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d5ed30-57ee-4d54-8402-7677917247d0" containerName="payload-processing"
Nov 03 13:35:13.485537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485495    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0ce7938f-84e4-4a81-b519-b100584149a3" containerName="revoke-keys"
Nov 03 13:35:13.485537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485500    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce7938f-84e4-4a81-b519-b100584149a3" containerName="revoke-keys"
Nov 03 13:35:13.485537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485506    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b2277d95-deaa-457e-aa15-5351aa31d91d" containerName="revoke-keys"
Nov 03 13:35:13.485537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485511    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2277d95-deaa-457e-aa15-5351aa31d91d" containerName="revoke-keys"
Nov 03 13:35:13.485708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485552    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b079a4df-8022-4fdb-b26d-34791a3523cb" containerName="payload-pre-processing"
Nov 03 13:35:13.485708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485557    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b079a4df-8022-4fdb-b26d-34791a3523cb" containerName="payload-pre-processing"
Nov 03 13:35:13.485708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485650    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b079a4df-8022-4fdb-b26d-34791a3523cb" containerName="payload-pre-processing"
Nov 03 13:35:13.485708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485658    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d5ed30-57ee-4d54-8402-7677917247d0" containerName="payload-processing"
Nov 03 13:35:13.485708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485670    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce7938f-84e4-4a81-b519-b100584149a3" containerName="revoke-keys"
Nov 03 13:35:13.485708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.485676    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2277d95-deaa-457e-aa15-5351aa31d91d" containerName="revoke-keys"
Nov 03 13:35:13.491960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.491924    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:13.508479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.508445    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"]
Nov 03 13:35:13.547466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.547398    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1ec4b892\""
Nov 03 13:35:13.547673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.547584    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1ec4b892-dockercfg-qwwlg\""
Nov 03 13:35:13.591246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.591182    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:13.614248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.611495    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94a1950-4189-4de8-9a4d-5a34ed885087-plugins-config-volume\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8\" (UID: \"e94a1950-4189-4de8-9a4d-5a34ed885087\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:13.614248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.611747    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkfpk\" (UniqueName: \"kubernetes.io/projected/e94a1950-4189-4de8-9a4d-5a34ed885087-kube-api-access-nkfpk\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8\" (UID: \"e94a1950-4189-4de8-9a4d-5a34ed885087\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:13.714274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.714222    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkfpk\" (UniqueName: \"kubernetes.io/projected/e94a1950-4189-4de8-9a4d-5a34ed885087-kube-api-access-nkfpk\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8\" (UID: \"e94a1950-4189-4de8-9a4d-5a34ed885087\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:13.714494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.714433    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94a1950-4189-4de8-9a4d-5a34ed885087-plugins-config-volume\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8\" (UID: \"e94a1950-4189-4de8-9a4d-5a34ed885087\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:13.715524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.715490    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94a1950-4189-4de8-9a4d-5a34ed885087-plugins-config-volume\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8\" (UID: \"e94a1950-4189-4de8-9a4d-5a34ed885087\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:13.727280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.727196    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkfpk\" (UniqueName: \"kubernetes.io/projected/e94a1950-4189-4de8-9a4d-5a34ed885087-kube-api-access-nkfpk\") pod \"payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8\" (UID: \"e94a1950-4189-4de8-9a4d-5a34ed885087\") " pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:13.814336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:13.814298    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:14.117516 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:14.117483    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"]
Nov 03 13:35:14.118518 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:14.118491    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94a1950_4189_4de8_9a4d_5a34ed885087.slice/crio-6ca20c11c5ab9785d68389c60a307f0031c706ec90fc99597f54a61aa452d6a0 WatchSource:0}: Error finding container 6ca20c11c5ab9785d68389c60a307f0031c706ec90fc99597f54a61aa452d6a0: Status 404 returned error can't find the container with id 6ca20c11c5ab9785d68389c60a307f0031c706ec90fc99597f54a61aa452d6a0
Nov 03 13:35:14.443143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:14.443054    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8" event={"ID":"e94a1950-4189-4de8-9a4d-5a34ed885087","Type":"ContainerStarted","Data":"71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af"}
Nov 03 13:35:14.443143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:14.443116    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:14.443143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:14.443140    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8" event={"ID":"e94a1950-4189-4de8-9a4d-5a34ed885087","Type":"ContainerStarted","Data":"6ca20c11c5ab9785d68389c60a307f0031c706ec90fc99597f54a61aa452d6a0"}
Nov 03 13:35:14.473126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:14.473079    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8" podStartSLOduration=1.473064463 podStartE2EDuration="1.473064463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:14.466433303 +0000 UTC m=+1555.544198331" watchObservedRunningTime="2026-11-03 13:35:14.473064463 +0000 UTC m=+1555.550829490"
Nov 03 13:35:14.518668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:14.518631    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:14.680274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:14.680241    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:15.386042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.386010    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"]
Nov 03 13:35:15.395450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.395413    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"]
Nov 03 13:35:15.395634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.395530    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.401695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.401494    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:35:15.401695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.401556    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:35:15.401695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.401595    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:35:15.401695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.401660    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:35:15.532602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.532538    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:15.542479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.542447    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlrnp\" (UniqueName: \"kubernetes.io/projected/f08a2cce-fd8a-4a32-9955-ebd5764e4814-kube-api-access-tlrnp\") pod \"maas-api-revoke-sub-a1fa54a0-r4nxc\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.542659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.542490    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1fa54a0-r4nxc\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.542659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.542615    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1fa54a0-r4nxc\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.644265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.644142    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlrnp\" (UniqueName: \"kubernetes.io/projected/f08a2cce-fd8a-4a32-9955-ebd5764e4814-kube-api-access-tlrnp\") pod \"maas-api-revoke-sub-a1fa54a0-r4nxc\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.644265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.644191    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1fa54a0-r4nxc\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.644559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.644305    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1fa54a0-r4nxc\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.645253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.645162    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1fa54a0-r4nxc\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.647451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.647427    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1fa54a0-r4nxc\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.653173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.653146    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlrnp\" (UniqueName: \"kubernetes.io/projected/f08a2cce-fd8a-4a32-9955-ebd5764e4814-kube-api-access-tlrnp\") pod \"maas-api-revoke-sub-a1fa54a0-r4nxc\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.712064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.712022    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:15.996355 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:15.996316    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08a2cce_fd8a_4a32_9955_ebd5764e4814.slice/crio-1f3cb1d69ac089267cd2a76e5adb5a9c2f0d87f5b6efbc5aede2590cf7d9b791 WatchSource:0}: Error finding container 1f3cb1d69ac089267cd2a76e5adb5a9c2f0d87f5b6efbc5aede2590cf7d9b791: Status 404 returned error can't find the container with id 1f3cb1d69ac089267cd2a76e5adb5a9c2f0d87f5b6efbc5aede2590cf7d9b791
Nov 03 13:35:15.998016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:15.997961    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"]
Nov 03 13:35:16.217385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:16.216376    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:16.465606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:16.465573    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc" event={"ID":"f08a2cce-fd8a-4a32-9955-ebd5764e4814","Type":"ContainerStarted","Data":"36a461aa398db81c24175f76fa560f8a6caee6d7746503f1568201d4db40a204"}
Nov 03 13:35:16.466038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:16.465659    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc" event={"ID":"f08a2cce-fd8a-4a32-9955-ebd5764e4814","Type":"ContainerStarted","Data":"1f3cb1d69ac089267cd2a76e5adb5a9c2f0d87f5b6efbc5aede2590cf7d9b791"}
Nov 03 13:35:16.484477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:16.484420    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc" podStartSLOduration=1.484406646 podStartE2EDuration="1.484406646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:16.48345254 +0000 UTC m=+1557.561217566" watchObservedRunningTime="2026-11-03 13:35:16.484406646 +0000 UTC m=+1557.562171669"
Nov 03 13:35:17.001714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:17.001685    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:17.223193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:17.223154    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:17.377517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:17.376982    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:17.455410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:17.455374    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"]
Nov 03 13:35:17.483164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:17.483126    5420 generic.go:356] "Generic (PLEG): container finished" podID="f08a2cce-fd8a-4a32-9955-ebd5764e4814" containerID="36a461aa398db81c24175f76fa560f8a6caee6d7746503f1568201d4db40a204" exitCode=0
Nov 03 13:35:17.483669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:17.483340    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc" event={"ID":"f08a2cce-fd8a-4a32-9955-ebd5764e4814","Type":"ContainerDied","Data":"36a461aa398db81c24175f76fa560f8a6caee6d7746503f1568201d4db40a204"}
Nov 03 13:35:17.500072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:17.500043    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:17.676336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:17.673638    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:17.918731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:17.918697    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:18.604755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:18.604662    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:18.826245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:18.823708    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:18.840076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:18.833608    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:18.993054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:18.992994    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:19.026649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.026558    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-xks-ca\") pod \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") "
Nov 03 13:35:19.026853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.026702    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08a2cce-fd8a-4a32-9955-ebd5764e4814-kube-api-access-tlrnp\" (UniqueName: \"kubernetes.io/projected/f08a2cce-fd8a-4a32-9955-ebd5764e4814-kube-api-access-tlrnp\") pod \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") "
Nov 03 13:35:19.026981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.026966    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-openshift-ca\") pod \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\" (UID: \"f08a2cce-fd8a-4a32-9955-ebd5764e4814\") "
Nov 03 13:35:19.027430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.027404    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-openshift-ca" pod "f08a2cce-fd8a-4a32-9955-ebd5764e4814" (UID: "f08a2cce-fd8a-4a32-9955-ebd5764e4814"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:19.027630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.027611    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:19.028887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.028863    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-xks-ca" pod "f08a2cce-fd8a-4a32-9955-ebd5764e4814" (UID: "f08a2cce-fd8a-4a32-9955-ebd5764e4814"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:19.029070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.029046    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08a2cce-fd8a-4a32-9955-ebd5764e4814-kube-api-access-tlrnp" pod "f08a2cce-fd8a-4a32-9955-ebd5764e4814" (UID: "f08a2cce-fd8a-4a32-9955-ebd5764e4814"). InnerVolumeSpecName "kube-api-access-tlrnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:19.129548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.129511    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08a2cce-fd8a-4a32-9955-ebd5764e4814-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:19.129548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.129542    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlrnp\" (UniqueName: \"kubernetes.io/projected/f08a2cce-fd8a-4a32-9955-ebd5764e4814-kube-api-access-tlrnp\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:19.503009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.502972    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"
Nov 03 13:35:19.503266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.502997    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc" event={"ID":"f08a2cce-fd8a-4a32-9955-ebd5764e4814","Type":"ContainerDied","Data":"1f3cb1d69ac089267cd2a76e5adb5a9c2f0d87f5b6efbc5aede2590cf7d9b791"}
Nov 03 13:35:19.503266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.503086    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f3cb1d69ac089267cd2a76e5adb5a9c2f0d87f5b6efbc5aede2590cf7d9b791"
Nov 03 13:35:19.738648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:19.738612    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:20.530927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:20.530894    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:21.264724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:21.264674    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:21.730693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:21.730657    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:21.811098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:21.811067    5420 scope.go:122] "RemoveContainer" containerID="a53bb3250ee1a63724cadca39fd52b9c43bfb7d0848eeba0332e8eeaf2130ba4"
Nov 03 13:35:21.822539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:21.822518    5420 scope.go:122] "RemoveContainer" containerID="85378d50b5c0d466315481245d10cd009684557e87d08d6e58edd2c41ab22b92"
Nov 03 13:35:21.832172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:21.832151    5420 scope.go:122] "RemoveContainer" containerID="88481cf8b016e585d0662218d97efce3a141c2c62570276d3c2b1ef7ca8a6ebe"
Nov 03 13:35:21.876710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:21.876680    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:22.290775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.290734    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:22.455708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.455669    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh" podUID="2f3425a2-8dc5-4666-bf51-b52abeb43601" containerName="payload-processing" containerID="cri-o://b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5" gracePeriod=25
Nov 03 13:35:22.481523 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:22.481492    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3425a2_8dc5_4666_bf51_b52abeb43601.slice/crio-b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:35:22.745877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.745847    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:35:22.787450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.787419    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-kube-api-access-wkdhj\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-kube-api-access-wkdhj\") pod \"2f3425a2-8dc5-4666-bf51-b52abeb43601\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") "
Nov 03 13:35:22.787643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.787489    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-plugins-config-volume\") pod \"2f3425a2-8dc5-4666-bf51-b52abeb43601\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") "
Nov 03 13:35:22.787643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.787555    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-routing-overlay-volume\") pod \"2f3425a2-8dc5-4666-bf51-b52abeb43601\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") "
Nov 03 13:35:22.787783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.787680    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-provider-credentials\") pod \"2f3425a2-8dc5-4666-bf51-b52abeb43601\" (UID: \"2f3425a2-8dc5-4666-bf51-b52abeb43601\") "
Nov 03 13:35:22.788039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.787965    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-routing-overlay-volume" pod "2f3425a2-8dc5-4666-bf51-b52abeb43601" (UID: "2f3425a2-8dc5-4666-bf51-b52abeb43601"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:22.788039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.788006    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-plugins-config-volume" pod "2f3425a2-8dc5-4666-bf51-b52abeb43601" (UID: "2f3425a2-8dc5-4666-bf51-b52abeb43601"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:22.788482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.788450    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:22.788482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.788481    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f3425a2-8dc5-4666-bf51-b52abeb43601-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:22.790574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.790550    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-provider-credentials" pod "2f3425a2-8dc5-4666-bf51-b52abeb43601" (UID: "2f3425a2-8dc5-4666-bf51-b52abeb43601"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:22.790667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.790619    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-kube-api-access-wkdhj" pod "2f3425a2-8dc5-4666-bf51-b52abeb43601" (UID: "2f3425a2-8dc5-4666-bf51-b52abeb43601"). InnerVolumeSpecName "kube-api-access-wkdhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:22.861693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.861610    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:22.898440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.897825    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:22.898440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:22.897861    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkdhj\" (UniqueName: \"kubernetes.io/projected/2f3425a2-8dc5-4666-bf51-b52abeb43601-kube-api-access-wkdhj\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:23.054221 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.054179    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:23.174541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.174469    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"]
Nov 03 13:35:23.175191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.175176    5420 cpu_manager.go:414] "containerMap: removing container" podUID="2f3425a2-8dc5-4666-bf51-b52abeb43601" containerName="payload-processing"
Nov 03 13:35:23.175248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.175193    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3425a2-8dc5-4666-bf51-b52abeb43601" containerName="payload-processing"
Nov 03 13:35:23.175248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.175228    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f08a2cce-fd8a-4a32-9955-ebd5764e4814" containerName="revoke-keys"
Nov 03 13:35:23.175248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.175235    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08a2cce-fd8a-4a32-9955-ebd5764e4814" containerName="revoke-keys"
Nov 03 13:35:23.175371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.175339    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3425a2-8dc5-4666-bf51-b52abeb43601" containerName="payload-processing"
Nov 03 13:35:23.175371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.175352    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08a2cce-fd8a-4a32-9955-ebd5764e4814" containerName="revoke-keys"
Nov 03 13:35:23.179470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.179454    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:23.189545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.189519    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c47670\""
Nov 03 13:35:23.189668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.189608    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c47670-dockercfg-9tc8h\""
Nov 03 13:35:23.194705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.194685    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"]
Nov 03 13:35:23.295232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.291778    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:23.321237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.306272    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f289deff-b01c-46c1-9336-bc43d9a8023b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c\" (UID: \"f289deff-b01c-46c1-9336-bc43d9a8023b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:23.321237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.306570    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zq8\" (UniqueName: \"kubernetes.io/projected/f289deff-b01c-46c1-9336-bc43d9a8023b-kube-api-access-p8zq8\") pod \"payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c\" (UID: \"f289deff-b01c-46c1-9336-bc43d9a8023b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:23.426782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.425280    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zq8\" (UniqueName: \"kubernetes.io/projected/f289deff-b01c-46c1-9336-bc43d9a8023b-kube-api-access-p8zq8\") pod \"payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c\" (UID: \"f289deff-b01c-46c1-9336-bc43d9a8023b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:23.426782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.425412    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f289deff-b01c-46c1-9336-bc43d9a8023b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c\" (UID: \"f289deff-b01c-46c1-9336-bc43d9a8023b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:23.426782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.426477    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f289deff-b01c-46c1-9336-bc43d9a8023b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c\" (UID: \"f289deff-b01c-46c1-9336-bc43d9a8023b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:23.458198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.456580    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zq8\" (UniqueName: \"kubernetes.io/projected/f289deff-b01c-46c1-9336-bc43d9a8023b-kube-api-access-p8zq8\") pod \"payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c\" (UID: \"f289deff-b01c-46c1-9336-bc43d9a8023b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:23.495565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.495536    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:23.552706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.552671    5420 generic.go:356] "Generic (PLEG): container finished" podID="2f3425a2-8dc5-4666-bf51-b52abeb43601" containerID="b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5" exitCode=0
Nov 03 13:35:23.552913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.552738    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"
Nov 03 13:35:23.553306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.553273    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh" event={"ID":"2f3425a2-8dc5-4666-bf51-b52abeb43601","Type":"ContainerDied","Data":"b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5"}
Nov 03 13:35:23.553437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.553336    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh" event={"ID":"2f3425a2-8dc5-4666-bf51-b52abeb43601","Type":"ContainerDied","Data":"4e01b6f66f6ca70c802e20766914f86137ddb4a707fc88b10e44a01b6d4ff2b3"}
Nov 03 13:35:23.553437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.553370    5420 scope.go:122] "RemoveContainer" containerID="b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5"
Nov 03 13:35:23.577589 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.577565    5420 scope.go:122] "RemoveContainer" containerID="b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5"
Nov 03 13:35:23.578009 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:23.577982    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5\": container with ID starting with b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5 not found: ID does not exist" containerID="b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5"
Nov 03 13:35:23.578114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.578015    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5"} err="failed to get container status \"b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5\": rpc error: code = NotFound desc = could not find container \"b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5\": container with ID starting with b89046c95b4a9580399c025a5a8766de8194f118b4180d547991b360872a8ad5 not found: ID does not exist"
Nov 03 13:35:23.599510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.599480    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"]
Nov 03 13:35:23.603774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.603749    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-84b5br2wh"]
Nov 03 13:35:23.786935 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:23.786901    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf289deff_b01c_46c1_9336_bc43d9a8023b.slice/crio-c66f1440d59bb576e4ed971f82b1e268d4dcae326ee1b7ee84fbfc93e06ad867 WatchSource:0}: Error finding container c66f1440d59bb576e4ed971f82b1e268d4dcae326ee1b7ee84fbfc93e06ad867: Status 404 returned error can't find the container with id c66f1440d59bb576e4ed971f82b1e268d4dcae326ee1b7ee84fbfc93e06ad867
Nov 03 13:35:23.791508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:23.791483    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"]
Nov 03 13:35:24.233750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:24.233705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:24.562492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:24.562408    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c" event={"ID":"f289deff-b01c-46c1-9336-bc43d9a8023b","Type":"ContainerStarted","Data":"1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2"}
Nov 03 13:35:24.562492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:24.562460    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c" event={"ID":"f289deff-b01c-46c1-9336-bc43d9a8023b","Type":"ContainerStarted","Data":"c66f1440d59bb576e4ed971f82b1e268d4dcae326ee1b7ee84fbfc93e06ad867"}
Nov 03 13:35:24.562950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:24.562618    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:24.595448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:24.595404    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c" podStartSLOduration=1.595391783 podStartE2EDuration="1.595391783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:24.594490618 +0000 UTC m=+1565.672255624" watchObservedRunningTime="2026-11-03 13:35:24.595391783 +0000 UTC m=+1565.673156846"
Nov 03 13:35:24.732835 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:24.732799    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:35:24.846798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:24.846702    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:25.086347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:25.086315    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3425a2-8dc5-4666-bf51-b52abeb43601" path="/var/lib/kubelet/pods/2f3425a2-8dc5-4666-bf51-b52abeb43601/volumes"
Nov 03 13:35:25.454943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:25.454916    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:35:25.651791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:25.651760    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:26.408509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.408464    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:26.532434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.532383    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"]
Nov 03 13:35:26.582299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.582267    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"]
Nov 03 13:35:26.582501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.582390    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.600473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.600442    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:26.677614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.677531    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.677614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.677572    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.677614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.677609    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.678059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.677667    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dgs\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-kube-api-access-c6dgs\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.779389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.779353    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.779565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.779408    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.779565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.779449    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.779565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.779478    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dgs\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-kube-api-access-c6dgs\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.780324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.780296    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.780475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.780376    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.782298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.782268    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.791865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.791838    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dgs\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-kube-api-access-c6dgs\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:26.924552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:26.924516    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:27.230012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.229958    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:27.373042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.373011    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"]
Nov 03 13:35:27.380458 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:27.380409    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9f0c17_2d2e_407d_aab9_10fd6c912e1d.slice/crio-e6466adf888d026496daaf3541cb18887d83d46dd56e43b4e0a5b2cdff59466e WatchSource:0}: Error finding container e6466adf888d026496daaf3541cb18887d83d46dd56e43b4e0a5b2cdff59466e: Status 404 returned error can't find the container with id e6466adf888d026496daaf3541cb18887d83d46dd56e43b4e0a5b2cdff59466e
Nov 03 13:35:27.433742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.433712    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:27.606303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.594959    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:27.660459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.659035    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" event={"ID":"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d","Type":"ContainerStarted","Data":"1db1b372e62ea8ef4df6c1badc3e1c59540780d393431b8171ae21271ef5ed8e"}
Nov 03 13:35:27.660459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.659101    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" event={"ID":"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d","Type":"ContainerStarted","Data":"e6466adf888d026496daaf3541cb18887d83d46dd56e43b4e0a5b2cdff59466e"}
Nov 03 13:35:27.660459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.660384    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:27.696033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.695867    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" podStartSLOduration=1.6958465999999999 podStartE2EDuration="1.6958466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:27.686811492 +0000 UTC m=+1568.764576518" watchObservedRunningTime="2026-11-03 13:35:27.6958466 +0000 UTC m=+1568.773611627"
Nov 03 13:35:27.710540 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:27.710198    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-ec4cc2d7b734: secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:27.710540 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:27.710267    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb: secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:27.710540 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:27.710342    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials podName:dd9f0c17-2d2e-407d-aab9-10fd6c912e1d nodeName:}" failed. No retries permitted until 2026-11-03 13:35:28.210326409 +0000 UTC m=+1569.288091414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" (UID: "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d") : secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:27.989150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.989117    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"]
Nov 03 13:35:27.994335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.994313    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:27.997538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.997517    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"]
Nov 03 13:35:27.998737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.998713    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:35:27.998862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.998748    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:35:27.998943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:27.998879    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:35:28.000058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.000039    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:35:28.014551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.014518    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcdp5\" (UniqueName: \"kubernetes.io/projected/0972fa32-c3eb-4522-89c1-700fbc38f167-kube-api-access-pcdp5\") pod \"maas-api-revoke-sub-18a137e9-plwcp\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.015553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.014577    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18a137e9-plwcp\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.015668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.015641    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18a137e9-plwcp\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.117601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.117557    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18a137e9-plwcp\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.120649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.120616    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcdp5\" (UniqueName: \"kubernetes.io/projected/0972fa32-c3eb-4522-89c1-700fbc38f167-kube-api-access-pcdp5\") pod \"maas-api-revoke-sub-18a137e9-plwcp\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.120904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.120881    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18a137e9-plwcp\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.123233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.122798    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18a137e9-plwcp\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.129877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.129848    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:28.131201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.131179    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18a137e9-plwcp\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.133679 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.133658    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcdp5\" (UniqueName: \"kubernetes.io/projected/0972fa32-c3eb-4522-89c1-700fbc38f167-kube-api-access-pcdp5\") pod \"maas-api-revoke-sub-18a137e9-plwcp\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.223355 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:28.223323    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-ec4cc2d7b734: secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:28.223355 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:28.223353    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb: secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:28.223554 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:28.223408    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials podName:dd9f0c17-2d2e-407d-aab9-10fd6c912e1d nodeName:}" failed. No retries permitted until 2026-11-03 13:35:29.223391831 +0000 UTC m=+1570.301156851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" (UID: "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d") : secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:28.283876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.283779    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"]
Nov 03 13:35:28.321269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.321235    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:28.323405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.323374    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"]
Nov 03 13:35:28.332607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.332582    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.337433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.337100    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"]
Nov 03 13:35:28.429921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.429889    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.430193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.430171    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.430381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.430363    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.430779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.430756    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gpcs\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-kube-api-access-9gpcs\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.532266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.532193    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gpcs\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-kube-api-access-9gpcs\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.532468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.532340    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.532468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.532394    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.532599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.532518    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.533077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.533052    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.533252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.533221    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.535106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.535058    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.541117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.541098    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gpcs\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-kube-api-access-9gpcs\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.652196 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:28.652166    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0972fa32_c3eb_4522_89c1_700fbc38f167.slice/crio-38bc88f486161192d99dbd60ee838a2c2f6eb03f57b06c0a1ed3a8ba035eaa96 WatchSource:0}: Error finding container 38bc88f486161192d99dbd60ee838a2c2f6eb03f57b06c0a1ed3a8ba035eaa96: Status 404 returned error can't find the container with id 38bc88f486161192d99dbd60ee838a2c2f6eb03f57b06c0a1ed3a8ba035eaa96
Nov 03 13:35:28.652753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.652671    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"]
Nov 03 13:35:28.667897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.667865    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp" event={"ID":"0972fa32-c3eb-4522-89c1-700fbc38f167","Type":"ContainerStarted","Data":"38bc88f486161192d99dbd60ee838a2c2f6eb03f57b06c0a1ed3a8ba035eaa96"}
Nov 03 13:35:28.686921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.686899    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:28.837873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.834104    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:28.975433 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:28.975404    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda718e7a8_6450_4c9d_875a_1fc972947dae.slice/crio-6f48b30c3026fdd10a4b56203ebb11d97acb2fe2ad4381f8e556a8957bb91003 WatchSource:0}: Error finding container 6f48b30c3026fdd10a4b56203ebb11d97acb2fe2ad4381f8e556a8957bb91003: Status 404 returned error can't find the container with id 6f48b30c3026fdd10a4b56203ebb11d97acb2fe2ad4381f8e556a8957bb91003
Nov 03 13:35:28.975916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:28.975894    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"]
Nov 03 13:35:29.249798 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:29.249770    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-ec4cc2d7b734: secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:29.249798 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:29.249795    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb: secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:29.249999 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:29.249847    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials podName:dd9f0c17-2d2e-407d-aab9-10fd6c912e1d nodeName:}" failed. No retries permitted until 2026-11-03 13:35:31.249831852 +0000 UTC m=+1572.327596856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" (UID: "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d") : secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:29.440384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:29.440344    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:29.677422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:29.677386    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz" event={"ID":"a718e7a8-6450-4c9d-875a-1fc972947dae","Type":"ContainerStarted","Data":"0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e"}
Nov 03 13:35:29.677604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:29.677457    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:29.677604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:29.677479    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz" event={"ID":"a718e7a8-6450-4c9d-875a-1fc972947dae","Type":"ContainerStarted","Data":"6f48b30c3026fdd10a4b56203ebb11d97acb2fe2ad4381f8e556a8957bb91003"}
Nov 03 13:35:29.679090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:29.679067    5420 generic.go:356] "Generic (PLEG): container finished" podID="0972fa32-c3eb-4522-89c1-700fbc38f167" containerID="c47e8b3a3d12a361b5e7c9dee13ca95b0ea62289b7b31cb0c11892ca3dcc6e36" exitCode=0
Nov 03 13:35:29.679170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:29.679129    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp" event={"ID":"0972fa32-c3eb-4522-89c1-700fbc38f167","Type":"ContainerDied","Data":"c47e8b3a3d12a361b5e7c9dee13ca95b0ea62289b7b31cb0c11892ca3dcc6e36"}
Nov 03 13:35:29.703354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:29.703311    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz" podStartSLOduration=1.703297666 podStartE2EDuration="1.703297666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:29.69728626 +0000 UTC m=+1570.775051288" watchObservedRunningTime="2026-11-03 13:35:29.703297666 +0000 UTC m=+1570.781062716"
Nov 03 13:35:30.210802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.210773    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:35:30.252087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.252053    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:30.867864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.867836    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:30.981572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.981542    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-xks-ca\") pod \"0972fa32-c3eb-4522-89c1-700fbc38f167\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") "
Nov 03 13:35:30.981774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.981595    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0972fa32-c3eb-4522-89c1-700fbc38f167-kube-api-access-pcdp5\" (UniqueName: \"kubernetes.io/projected/0972fa32-c3eb-4522-89c1-700fbc38f167-kube-api-access-pcdp5\") pod \"0972fa32-c3eb-4522-89c1-700fbc38f167\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") "
Nov 03 13:35:30.981850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.981771    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-openshift-ca\") pod \"0972fa32-c3eb-4522-89c1-700fbc38f167\" (UID: \"0972fa32-c3eb-4522-89c1-700fbc38f167\") "
Nov 03 13:35:30.982146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.982097    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-openshift-ca" pod "0972fa32-c3eb-4522-89c1-700fbc38f167" (UID: "0972fa32-c3eb-4522-89c1-700fbc38f167"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:30.982483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.982465    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:30.983734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.983711    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-xks-ca" pod "0972fa32-c3eb-4522-89c1-700fbc38f167" (UID: "0972fa32-c3eb-4522-89c1-700fbc38f167"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:30.983838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:30.983809    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0972fa32-c3eb-4522-89c1-700fbc38f167-kube-api-access-pcdp5" pod "0972fa32-c3eb-4522-89c1-700fbc38f167" (UID: "0972fa32-c3eb-4522-89c1-700fbc38f167"). InnerVolumeSpecName "kube-api-access-pcdp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:31.084069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:31.084039    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcdp5\" (UniqueName: \"kubernetes.io/projected/0972fa32-c3eb-4522-89c1-700fbc38f167-kube-api-access-pcdp5\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:31.084069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:31.084066    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0972fa32-c3eb-4522-89c1-700fbc38f167-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:31.136546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:31.136519    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:31.292293 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:31.291676    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-ec4cc2d7b734: secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:31.292293 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:31.291712    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb: secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:31.292293 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:31.291770    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials podName:dd9f0c17-2d2e-407d-aab9-10fd6c912e1d nodeName:}" failed. No retries permitted until 2026-11-03 13:35:35.291752328 +0000 UTC m=+1576.369517349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" (UID: "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d") : secret "agc-cred-ec4cc2d7b734" not found
Nov 03 13:35:31.778241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:31.777269    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp" event={"ID":"0972fa32-c3eb-4522-89c1-700fbc38f167","Type":"ContainerDied","Data":"38bc88f486161192d99dbd60ee838a2c2f6eb03f57b06c0a1ed3a8ba035eaa96"}
Nov 03 13:35:31.778241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:31.777331    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38bc88f486161192d99dbd60ee838a2c2f6eb03f57b06c0a1ed3a8ba035eaa96"
Nov 03 13:35:31.778241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:31.777411    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"
Nov 03 13:35:32.421900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:32.421848    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:33.581376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:33.581333    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:33.944297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:33.942990    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:34.685235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:34.680289    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" podUID="dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" containerName="payload-processing" containerID="cri-o://1db1b372e62ea8ef4df6c1badc3e1c59540780d393431b8171ae21271ef5ed8e" gracePeriod=25
Nov 03 13:35:34.860738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:34.860704    5420 generic.go:356] "Generic (PLEG): container finished" podID="dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" containerID="1db1b372e62ea8ef4df6c1badc3e1c59540780d393431b8171ae21271ef5ed8e" exitCode=0
Nov 03 13:35:34.860938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:34.860842    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" event={"ID":"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d","Type":"ContainerDied","Data":"1db1b372e62ea8ef4df6c1badc3e1c59540780d393431b8171ae21271ef5ed8e"}
Nov 03 13:35:34.884401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:34.883533    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:35.033027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.032999    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:35.142503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.142474    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-routing-overlay-volume\") pod \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") "
Nov 03 13:35:35.142681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.142555    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-kube-api-access-c6dgs\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-kube-api-access-c6dgs\") pod \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") "
Nov 03 13:35:35.142681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.142619    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials\") pod \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") "
Nov 03 13:35:35.142681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.142642    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-plugins-config-volume\") pod \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\" (UID: \"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d\") "
Nov 03 13:35:35.142920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.142892    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-routing-overlay-volume" pod "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" (UID: "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:35.143169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.143144    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-plugins-config-volume" pod "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" (UID: "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:35.143376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.143360    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:35.143424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.143379    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:35.144728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.144700    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials" pod "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" (UID: "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:35.144834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.144810    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-kube-api-access-c6dgs" pod "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" (UID: "dd9f0c17-2d2e-407d-aab9-10fd6c912e1d"). InnerVolumeSpecName "kube-api-access-c6dgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:35.245272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.245170    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6dgs\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-kube-api-access-c6dgs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:35.245272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.245202    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:35.575034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.574947    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:35:35.732258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.729294    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:35.872047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.871958    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"
Nov 03 13:35:35.872047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.871995    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb" event={"ID":"dd9f0c17-2d2e-407d-aab9-10fd6c912e1d","Type":"ContainerDied","Data":"e6466adf888d026496daaf3541cb18887d83d46dd56e43b4e0a5b2cdff59466e"}
Nov 03 13:35:35.872306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.872082    5420 scope.go:122] "RemoveContainer" containerID="1db1b372e62ea8ef4df6c1badc3e1c59540780d393431b8171ae21271ef5ed8e"
Nov 03 13:35:35.899589 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.899552    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"]
Nov 03 13:35:35.906764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:35.906735    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-56d4vlsrb"]
Nov 03 13:35:36.528763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:36.528732    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:36.670922 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:36.670888    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:36.883739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:36.883661    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:37.087018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:37.086983    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" path="/var/lib/kubelet/pods/dd9f0c17-2d2e-407d-aab9-10fd6c912e1d/volumes"
Nov 03 13:35:37.369943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:37.368199    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:38.142264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:38.140865    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:38.808330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:38.808027    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:39.538039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:39.537972    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:40.046301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.046267    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:40.169512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.169485    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"]
Nov 03 13:35:40.173125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.173100    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:40.206704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.206669    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:40.210130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.210106    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"]
Nov 03 13:35:40.211030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.211010    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0972fa32-c3eb-4522-89c1-700fbc38f167" containerName="revoke-keys"
Nov 03 13:35:40.211030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.211030    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0972fa32-c3eb-4522-89c1-700fbc38f167" containerName="revoke-keys"
Nov 03 13:35:40.211252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.211050    5420 cpu_manager.go:414] "containerMap: removing container" podUID="dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" containerName="payload-processing"
Nov 03 13:35:40.211252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.211058    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" containerName="payload-processing"
Nov 03 13:35:40.211380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.211271    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0972fa32-c3eb-4522-89c1-700fbc38f167" containerName="revoke-keys"
Nov 03 13:35:40.211380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.211294    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9f0c17-2d2e-407d-aab9-10fd6c912e1d" containerName="payload-processing"
Nov 03 13:35:40.216901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.216877    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.268448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.268417    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"]
Nov 03 13:35:40.328631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.328554    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmvgg\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-kube-api-access-vmvgg\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.328791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.328734    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.328791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.328775    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.328901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.328889    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.430460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.430420    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.430647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.430481    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.430647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.430578    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.430788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.430717    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmvgg\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-kube-api-access-vmvgg\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.431407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.431380    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.431548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.431443    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.433613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.433582    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.439740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.439715    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmvgg\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-kube-api-access-vmvgg\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.512645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.512611    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:40.562233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.560520    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:40.646568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.645462    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"]
Nov 03 13:35:40.654907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.654469    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.666276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.658478    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"]
Nov 03 13:35:40.666276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.660824    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:35:40.666276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.661798    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:35:40.666276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.661858    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:35:40.666276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.662130    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:35:40.751737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.750558    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc66dcf4-nq6cr\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.751737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.750611    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4dls\" (UniqueName: \"kubernetes.io/projected/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-kube-api-access-v4dls\") pod \"maas-api-revoke-sub-dc66dcf4-nq6cr\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.751737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.750767    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc66dcf4-nq6cr\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.851950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.851916    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc66dcf4-nq6cr\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.851950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.851954    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4dls\" (UniqueName: \"kubernetes.io/projected/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-kube-api-access-v4dls\") pod \"maas-api-revoke-sub-dc66dcf4-nq6cr\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.852149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.852055    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc66dcf4-nq6cr\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.852785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.852763    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc66dcf4-nq6cr\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.854333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.854312    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc66dcf4-nq6cr\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.861279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.861259    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4dls\" (UniqueName: \"kubernetes.io/projected/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-kube-api-access-v4dls\") pod \"maas-api-revoke-sub-dc66dcf4-nq6cr\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:40.964802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.964769    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"]
Nov 03 13:35:40.965026 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:40.965001    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41763fbe_38f6_4519_becd_ce0e6f7c5110.slice/crio-e6f8a152c1669a36246b811f8de1038a8cb60907fea5b84b4fa82c15069eaded WatchSource:0}: Error finding container e6f8a152c1669a36246b811f8de1038a8cb60907fea5b84b4fa82c15069eaded: Status 404 returned error can't find the container with id e6f8a152c1669a36246b811f8de1038a8cb60907fea5b84b4fa82c15069eaded
Nov 03 13:35:40.990269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:40.990246    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:41.197273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:41.195364    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:41.278932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:41.278903    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"]
Nov 03 13:35:41.280661 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:41.280633    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3497c5_1eea_4841_9a2e_c64ccf7e571b.slice/crio-f1524be1bf0542c66965e67a38bc8a677753b45c39c62b60cf9fcde08ea52474 WatchSource:0}: Error finding container f1524be1bf0542c66965e67a38bc8a677753b45c39c62b60cf9fcde08ea52474: Status 404 returned error can't find the container with id f1524be1bf0542c66965e67a38bc8a677753b45c39c62b60cf9fcde08ea52474
Nov 03 13:35:41.929531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:41.929382    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8" event={"ID":"41763fbe-38f6-4519-becd-ce0e6f7c5110","Type":"ContainerStarted","Data":"89fb0bc470cf7b4e099890ebd06f7bb85bb11e223679c4a3128428fd3d0ae07f"}
Nov 03 13:35:41.930018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:41.929802    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:41.930018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:41.929843    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8" event={"ID":"41763fbe-38f6-4519-becd-ce0e6f7c5110","Type":"ContainerStarted","Data":"e6f8a152c1669a36246b811f8de1038a8cb60907fea5b84b4fa82c15069eaded"}
Nov 03 13:35:41.931630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:41.931599    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr" event={"ID":"6a3497c5-1eea-4841-9a2e-c64ccf7e571b","Type":"ContainerStarted","Data":"3b9507868d0b3669ceb5222ad6dc67dbb6270f37f9492ed0582658332cbce7d1"}
Nov 03 13:35:41.931762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:41.931682    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr" event={"ID":"6a3497c5-1eea-4841-9a2e-c64ccf7e571b","Type":"ContainerStarted","Data":"f1524be1bf0542c66965e67a38bc8a677753b45c39c62b60cf9fcde08ea52474"}
Nov 03 13:35:41.957868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:41.957811    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8" podStartSLOduration=1.957796563 podStartE2EDuration="1.957796563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:41.954019162 +0000 UTC m=+1583.031784189" watchObservedRunningTime="2026-11-03 13:35:41.957796563 +0000 UTC m=+1583.035561588"
Nov 03 13:35:41.976600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:41.976534    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr" podStartSLOduration=1.976513401 podStartE2EDuration="1.976513401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:41.97175643 +0000 UTC m=+1583.049521457" watchObservedRunningTime="2026-11-03 13:35:41.976513401 +0000 UTC m=+1583.054278427"
Nov 03 13:35:42.942549 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:42.942517    5420 generic.go:356] "Generic (PLEG): container finished" podID="6a3497c5-1eea-4841-9a2e-c64ccf7e571b" containerID="3b9507868d0b3669ceb5222ad6dc67dbb6270f37f9492ed0582658332cbce7d1" exitCode=0
Nov 03 13:35:42.943026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:42.942602    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr" event={"ID":"6a3497c5-1eea-4841-9a2e-c64ccf7e571b","Type":"ContainerDied","Data":"3b9507868d0b3669ceb5222ad6dc67dbb6270f37f9492ed0582658332cbce7d1"}
Nov 03 13:35:42.998835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:42.998806    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:43.128255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:43.128188    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:43.307647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:43.307618    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:43.459771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:43.459735    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:43.690676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:43.690587    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:44.005426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.005394    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:44.147401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.147372    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:44.226291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.205594    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:44.334307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.333403    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-kube-api-access-v4dls\" (UniqueName: \"kubernetes.io/projected/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-kube-api-access-v4dls\") pod \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") "
Nov 03 13:35:44.334307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.333820    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-openshift-ca\") pod \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") "
Nov 03 13:35:44.334307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.333862    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-xks-ca\") pod \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\" (UID: \"6a3497c5-1eea-4841-9a2e-c64ccf7e571b\") "
Nov 03 13:35:44.334891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.334570    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-openshift-ca" pod "6a3497c5-1eea-4841-9a2e-c64ccf7e571b" (UID: "6a3497c5-1eea-4841-9a2e-c64ccf7e571b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:44.335611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.335465    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:44.356433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.355558    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-xks-ca" pod "6a3497c5-1eea-4841-9a2e-c64ccf7e571b" (UID: "6a3497c5-1eea-4841-9a2e-c64ccf7e571b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:35:44.356433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.355699    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-kube-api-access-v4dls" pod "6a3497c5-1eea-4841-9a2e-c64ccf7e571b" (UID: "6a3497c5-1eea-4841-9a2e-c64ccf7e571b"). InnerVolumeSpecName "kube-api-access-v4dls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:44.357179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.357152    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:44.437573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.437534    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:44.437573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.437575    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4dls\" (UniqueName: \"kubernetes.io/projected/6a3497c5-1eea-4841-9a2e-c64ccf7e571b-kube-api-access-v4dls\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:44.683435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.683351    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:44.866579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.866547    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:44.962904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.962873    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr" event={"ID":"6a3497c5-1eea-4841-9a2e-c64ccf7e571b","Type":"ContainerDied","Data":"f1524be1bf0542c66965e67a38bc8a677753b45c39c62b60cf9fcde08ea52474"}
Nov 03 13:35:44.963130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.962924    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1524be1bf0542c66965e67a38bc8a677753b45c39c62b60cf9fcde08ea52474"
Nov 03 13:35:44.963130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:44.962984    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"
Nov 03 13:35:45.170800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.170758    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz" podUID="a718e7a8-6450-4c9d-875a-1fc972947dae" containerName="payload-processing" containerID="cri-o://0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e" gracePeriod=25
Nov 03 13:35:45.449429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.449401    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:45.563535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.563444    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-routing-overlay-volume\") pod \"a718e7a8-6450-4c9d-875a-1fc972947dae\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") "
Nov 03 13:35:45.563722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.563552    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-kube-api-access-9gpcs\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-kube-api-access-9gpcs\") pod \"a718e7a8-6450-4c9d-875a-1fc972947dae\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") "
Nov 03 13:35:45.563722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.563711    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-provider-credentials\") pod \"a718e7a8-6450-4c9d-875a-1fc972947dae\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") "
Nov 03 13:35:45.563925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.563806    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-routing-overlay-volume" pod "a718e7a8-6450-4c9d-875a-1fc972947dae" (UID: "a718e7a8-6450-4c9d-875a-1fc972947dae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:45.563925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.563837    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-plugins-config-volume\") pod \"a718e7a8-6450-4c9d-875a-1fc972947dae\" (UID: \"a718e7a8-6450-4c9d-875a-1fc972947dae\") "
Nov 03 13:35:45.564194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.564173    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-plugins-config-volume" pod "a718e7a8-6450-4c9d-875a-1fc972947dae" (UID: "a718e7a8-6450-4c9d-875a-1fc972947dae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:45.564500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.564482    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:45.564574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.564506    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a718e7a8-6450-4c9d-875a-1fc972947dae-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:45.565798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.565776    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-provider-credentials" pod "a718e7a8-6450-4c9d-875a-1fc972947dae" (UID: "a718e7a8-6450-4c9d-875a-1fc972947dae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:45.565899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.565803    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-kube-api-access-9gpcs" pod "a718e7a8-6450-4c9d-875a-1fc972947dae" (UID: "a718e7a8-6450-4c9d-875a-1fc972947dae"). InnerVolumeSpecName "kube-api-access-9gpcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:45.625838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.625802    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:45.666480 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.666447    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:45.666480 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.666481    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gpcs\" (UniqueName: \"kubernetes.io/projected/a718e7a8-6450-4c9d-875a-1fc972947dae-kube-api-access-9gpcs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:45.974895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.974858    5420 generic.go:356] "Generic (PLEG): container finished" podID="a718e7a8-6450-4c9d-875a-1fc972947dae" containerID="0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e" exitCode=0
Nov 03 13:35:45.975076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.974914    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"
Nov 03 13:35:45.975076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.974943    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz" event={"ID":"a718e7a8-6450-4c9d-875a-1fc972947dae","Type":"ContainerDied","Data":"0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e"}
Nov 03 13:35:45.975076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.975007    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz" event={"ID":"a718e7a8-6450-4c9d-875a-1fc972947dae","Type":"ContainerDied","Data":"6f48b30c3026fdd10a4b56203ebb11d97acb2fe2ad4381f8e556a8957bb91003"}
Nov 03 13:35:45.975076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.975038    5420 scope.go:122] "RemoveContainer" containerID="0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e"
Nov 03 13:35:45.987048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.987031    5420 scope.go:122] "RemoveContainer" containerID="0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e"
Nov 03 13:35:45.987323 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:45.987303    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e\": container with ID starting with 0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e not found: ID does not exist" containerID="0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e"
Nov 03 13:35:45.987390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:45.987337    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e"} err="failed to get container status \"0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e\": rpc error: code = NotFound desc = could not find container \"0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e\": container with ID starting with 0347a4677f8e567c6d816db530c42a852c15e1b36da8f8d7226b5e3aa76bae2e not found: ID does not exist"
Nov 03 13:35:46.001894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:46.001867    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"]
Nov 03 13:35:46.006712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:46.006685    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554krpjz"]
Nov 03 13:35:46.229105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:46.228548    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:46.824026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:46.823985    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:47.086132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:47.086054    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a718e7a8-6450-4c9d-875a-1fc972947dae" path="/var/lib/kubelet/pods/a718e7a8-6450-4c9d-875a-1fc972947dae/volumes"
Nov 03 13:35:47.618859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:47.618817    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:48.460577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:48.460546    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:48.648639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:48.648581    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:49.554119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.554084    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:49.880846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.880765    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:49.914600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.914570    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"]
Nov 03 13:35:49.915264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.915249    5420 cpu_manager.go:414] "containerMap: removing container" podUID="a718e7a8-6450-4c9d-875a-1fc972947dae" containerName="payload-processing"
Nov 03 13:35:49.915327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.915266    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a718e7a8-6450-4c9d-875a-1fc972947dae" containerName="payload-processing"
Nov 03 13:35:49.915327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.915310    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6a3497c5-1eea-4841-9a2e-c64ccf7e571b" containerName="revoke-keys"
Nov 03 13:35:49.915327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.915315    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3497c5-1eea-4841-9a2e-c64ccf7e571b" containerName="revoke-keys"
Nov 03 13:35:49.915431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.915415    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a718e7a8-6450-4c9d-875a-1fc972947dae" containerName="payload-processing"
Nov 03 13:35:49.915470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.915434    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3497c5-1eea-4841-9a2e-c64ccf7e571b" containerName="revoke-keys"
Nov 03 13:35:49.920766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.920746    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:35:49.928056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.928033    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7e439d04\""
Nov 03 13:35:49.935669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.935644    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"]
Nov 03 13:35:49.965137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.965103    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"]
Nov 03 13:35:49.985096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.985060    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:35:49.991245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:49.991202    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7e439d04-dockercfg-hclz9\""
Nov 03 13:35:50.003265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.003240    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"]
Nov 03 13:35:50.060487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.060439    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae77e87-e96d-45d4-a495-7dc77e619250-plugins-config-volume\") pod \"payload-processing-e2e-mt-7e439d04-588c697959-8dgrg\" (UID: \"bae77e87-e96d-45d4-a495-7dc77e619250\") " pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:35:50.061065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.061018    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdfw\" (UniqueName: \"kubernetes.io/projected/bae77e87-e96d-45d4-a495-7dc77e619250-kube-api-access-dkdfw\") pod \"payload-processing-e2e-mt-7e439d04-588c697959-8dgrg\" (UID: \"bae77e87-e96d-45d4-a495-7dc77e619250\") " pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:35:50.061326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.061303    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6c86e0-9ece-4256-8456-8879c6782878-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h\" (UID: \"4b6c86e0-9ece-4256-8456-8879c6782878\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:35:50.061485 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.061465    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfxp\" (UniqueName: \"kubernetes.io/projected/4b6c86e0-9ece-4256-8456-8879c6782878-kube-api-access-hcfxp\") pod \"payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h\" (UID: \"4b6c86e0-9ece-4256-8456-8879c6782878\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:35:50.096389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.096350    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:50.163724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.163633    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdfw\" (UniqueName: \"kubernetes.io/projected/bae77e87-e96d-45d4-a495-7dc77e619250-kube-api-access-dkdfw\") pod \"payload-processing-e2e-mt-7e439d04-588c697959-8dgrg\" (UID: \"bae77e87-e96d-45d4-a495-7dc77e619250\") " pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:35:50.163890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.163736    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6c86e0-9ece-4256-8456-8879c6782878-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h\" (UID: \"4b6c86e0-9ece-4256-8456-8879c6782878\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:35:50.163890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.163764    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfxp\" (UniqueName: \"kubernetes.io/projected/4b6c86e0-9ece-4256-8456-8879c6782878-kube-api-access-hcfxp\") pod \"payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h\" (UID: \"4b6c86e0-9ece-4256-8456-8879c6782878\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:35:50.163983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.163957    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae77e87-e96d-45d4-a495-7dc77e619250-plugins-config-volume\") pod \"payload-processing-e2e-mt-7e439d04-588c697959-8dgrg\" (UID: \"bae77e87-e96d-45d4-a495-7dc77e619250\") " pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:35:50.165886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.165858    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae77e87-e96d-45d4-a495-7dc77e619250-plugins-config-volume\") pod \"payload-processing-e2e-mt-7e439d04-588c697959-8dgrg\" (UID: \"bae77e87-e96d-45d4-a495-7dc77e619250\") " pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:35:50.167027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.167003    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6c86e0-9ece-4256-8456-8879c6782878-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h\" (UID: \"4b6c86e0-9ece-4256-8456-8879c6782878\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:35:50.177913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.177883    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdfw\" (UniqueName: \"kubernetes.io/projected/bae77e87-e96d-45d4-a495-7dc77e619250-kube-api-access-dkdfw\") pod \"payload-processing-e2e-mt-7e439d04-588c697959-8dgrg\" (UID: \"bae77e87-e96d-45d4-a495-7dc77e619250\") " pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:35:50.179184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.179153    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfxp\" (UniqueName: \"kubernetes.io/projected/4b6c86e0-9ece-4256-8456-8879c6782878-kube-api-access-hcfxp\") pod \"payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h\" (UID: \"4b6c86e0-9ece-4256-8456-8879c6782878\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:35:50.239175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.239142    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:35:50.308266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.307763    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:35:50.525528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.519464    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:50.571528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.570443    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"]
Nov 03 13:35:50.634774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:50.634732    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"]
Nov 03 13:35:50.634915 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:50.634871    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae77e87_e96d_45d4_a495_7dc77e619250.slice/crio-24ad0459ac0f5e225698816926ee2adb77c0da35e2da920b7cff97c49547f987 WatchSource:0}: Error finding container 24ad0459ac0f5e225698816926ee2adb77c0da35e2da920b7cff97c49547f987: Status 404 returned error can't find the container with id 24ad0459ac0f5e225698816926ee2adb77c0da35e2da920b7cff97c49547f987
Nov 03 13:35:51.134057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.134001    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:35:51.134709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.134689    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:35:51.134967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.134932    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg" event={"ID":"bae77e87-e96d-45d4-a495-7dc77e619250","Type":"ContainerStarted","Data":"41196dcba1b59bf70ac9ff3736e59f9745e308dc8d043bb00a1c60ad74123d5f"}
Nov 03 13:35:51.135141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.135107    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg" event={"ID":"bae77e87-e96d-45d4-a495-7dc77e619250","Type":"ContainerStarted","Data":"24ad0459ac0f5e225698816926ee2adb77c0da35e2da920b7cff97c49547f987"}
Nov 03 13:35:51.135296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.135279    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h" event={"ID":"4b6c86e0-9ece-4256-8456-8879c6782878","Type":"ContainerStarted","Data":"a28d98170c8a25070fa2f596a848dc13dba94eb563b642d97567239d66c185c1"}
Nov 03 13:35:51.135448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.135432    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h" event={"ID":"4b6c86e0-9ece-4256-8456-8879c6782878","Type":"ContainerStarted","Data":"1865d9b06f60127a33967b8d0bd3288ac0a6ba85ba9168f77099502f467640ce"}
Nov 03 13:35:51.156604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.156561    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg" podStartSLOduration=2.156542522 podStartE2EDuration="2.156542522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:51.139621269 +0000 UTC m=+1592.217386296" watchObservedRunningTime="2026-11-03 13:35:51.156542522 +0000 UTC m=+1592.234307549"
Nov 03 13:35:51.174674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.174624    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h" podStartSLOduration=2.174610655 podStartE2EDuration="2.174610655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:51.171448954 +0000 UTC m=+1592.249213978" watchObservedRunningTime="2026-11-03 13:35:51.174610655 +0000 UTC m=+1592.252375680"
Nov 03 13:35:51.273902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.273871    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:51.980264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:51.969842    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:52.081268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:52.081235    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"]
Nov 03 13:35:52.085166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:52.085140    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:52.089800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:52.089749    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"]
Nov 03 13:35:52.126475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:52.126440    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:52.271692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:52.270790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:53.059563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.059528    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:53.122288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.122257    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"]
Nov 03 13:35:53.127023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.127004    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.135074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.135042    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-e942f4e7c82f\""
Nov 03 13:35:53.135074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.135058    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"payload-processing-external-model-e2e--f52a173c-dockercfg-8mjkq\""
Nov 03 13:35:53.142598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.142574    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"]
Nov 03 13:35:53.229448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.226200    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:53.281311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.280816    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.281311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.280938    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk5fz\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-kube-api-access-jk5fz\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.281311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.281051    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.281311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.281161    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.383233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.383072    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.383432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.383365    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.383508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.383435    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5fz\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-kube-api-access-jk5fz\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.383581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.383516    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.384316 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:53.384291    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found
Nov 03 13:35:53.384462 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:53.384450    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4: secret "agc-cred-e942f4e7c82f" not found
Nov 03 13:35:53.384634 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:53.384613    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials podName:d1c11b1a-1635-474c-bbdf-0860aeddf84b nodeName:}" failed. No retries permitted until 2026-11-03 13:35:53.884590644 +0000 UTC m=+1594.962355665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4" (UID: "d1c11b1a-1635-474c-bbdf-0860aeddf84b") : secret "agc-cred-e942f4e7c82f" not found
Nov 03 13:35:53.385302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.384423    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.386270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.385606    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.397472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.396800    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5fz\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-kube-api-access-jk5fz\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.402844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.402819    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:53.405300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.405253    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"]
Nov 03 13:35:53.405766 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:53.405743    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4" podUID="d1c11b1a-1635-474c-bbdf-0860aeddf84b"
Nov 03 13:35:53.588160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.588129    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:53.784642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.784613    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"]
Nov 03 13:35:53.784945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.784926    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:53.790469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.790452    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.799227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.799186    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-9f71f2-openshift-default-dockercfg-mxb5p\""
Nov 03 13:35:53.809512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.809481    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"]
Nov 03 13:35:53.890323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890279    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-certs\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.890503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890352    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrx48\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-kube-api-access-vrx48\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.890548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890512    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-envoy\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.890594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890569    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-credential-socket\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.890682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890660    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52845948-0621-4d3a-895d-b57495762003-istio-podinfo\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.890738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890690    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-istio-token\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.890738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890732    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:53.890826 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:53.890816    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found
Nov 03 13:35:53.890870 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:53.890827    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4: secret "agc-cred-e942f4e7c82f" not found
Nov 03 13:35:53.890870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890832    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-data\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.890870 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:53.890867    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials podName:d1c11b1a-1635-474c-bbdf-0860aeddf84b nodeName:}" failed. No retries permitted until 2026-11-03 13:35:54.890854948 +0000 UTC m=+1595.968619951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4" (UID: "d1c11b1a-1635-474c-bbdf-0860aeddf84b") : secret "agc-cred-e942f4e7c82f" not found
Nov 03 13:35:53.890989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890934    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52845948-0621-4d3a-895d-b57495762003-istiod-ca-cert\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.890989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.890959    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-socket\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992223 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992171    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-credential-socket\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992264    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52845948-0621-4d3a-895d-b57495762003-istio-podinfo\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992299    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-istio-token\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992390    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-data\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992489    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52845948-0621-4d3a-895d-b57495762003-istiod-ca-cert\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992525    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-socket\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992566    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-certs\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992621    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrx48\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-kube-api-access-vrx48\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992646    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-credential-socket\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.992890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.992809    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-envoy\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.994503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.993545    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-socket\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.994503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.994057    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-data\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.994503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.994107    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52845948-0621-4d3a-895d-b57495762003-istiod-ca-cert\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.994503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.994147    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-certs\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:53.996552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:53.996117    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52845948-0621-4d3a-895d-b57495762003-istio-podinfo\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:54.035282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.012830    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-envoy\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:54.053236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.047604    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-istio-token\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:54.053236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.052039    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrx48\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-kube-api-access-vrx48\") pod \"e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") " pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:54.100232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.096671    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:54.112780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.112750    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:54.183929 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.183898    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:54.205377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.205295    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:54.303147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.303064    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-plugins-config-volume\") pod \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") "
Nov 03 13:35:54.303590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.303343    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-kube-api-access-jk5fz\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-kube-api-access-jk5fz\") pod \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") "
Nov 03 13:35:54.303590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.303404    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-routing-overlay-volume\") pod \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") "
Nov 03 13:35:54.304597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.304570    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-plugins-config-volume" pod "d1c11b1a-1635-474c-bbdf-0860aeddf84b" (UID: "d1c11b1a-1635-474c-bbdf-0860aeddf84b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:54.304729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.304590    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-routing-overlay-volume" pod "d1c11b1a-1635-474c-bbdf-0860aeddf84b" (UID: "d1c11b1a-1635-474c-bbdf-0860aeddf84b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:54.306994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.306950    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-kube-api-access-jk5fz" pod "d1c11b1a-1635-474c-bbdf-0860aeddf84b" (UID: "d1c11b1a-1635-474c-bbdf-0860aeddf84b"). InnerVolumeSpecName "kube-api-access-jk5fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:54.406045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.406003    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk5fz\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-kube-api-access-jk5fz\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:54.406045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.406040    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:54.406335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.406056    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c11b1a-1635-474c-bbdf-0860aeddf84b-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:54.480960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.480926    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:54.495905 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:54.495850    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52845948_0621_4d3a_895d_b57495762003.slice/crio-1800b7a4f282949b10a87dd02577ebbaaf8f504f0ed5ec47a61116b86abd8d01 WatchSource:0}: Error finding container 1800b7a4f282949b10a87dd02577ebbaaf8f504f0ed5ec47a61116b86abd8d01: Status 404 returned error can't find the container with id 1800b7a4f282949b10a87dd02577ebbaaf8f504f0ed5ec47a61116b86abd8d01
Nov 03 13:35:54.496830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.496738    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"]
Nov 03 13:35:54.499912 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.499868    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:35:54.500070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.499965    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:35:54.500070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.500017    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:35:54.733525 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:54.733492    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:35:54.915050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:54.914970    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4\" (UID: \"d1c11b1a-1635-474c-bbdf-0860aeddf84b\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:54.915237 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:54.915123    5420 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found
Nov 03 13:35:54.915237 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:54.915138    5420 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4: secret "agc-cred-e942f4e7c82f" not found
Nov 03 13:35:54.915237 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:35:54.915191    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials podName:d1c11b1a-1635-474c-bbdf-0860aeddf84b nodeName:}" failed. No retries permitted until 2026-11-03 13:35:56.915175711 +0000 UTC m=+1597.992940722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4" (UID: "d1c11b1a-1635-474c-bbdf-0860aeddf84b") : secret "agc-cred-e942f4e7c82f" not found
Nov 03 13:35:55.114015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:55.113982    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:55.193841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:55.193771    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"
Nov 03 13:35:55.193981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:55.193773    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" event={"ID":"52845948-0621-4d3a-895d-b57495762003","Type":"ContainerStarted","Data":"2888860214ac067a034256b03f867aed3bf06663f415604f7c3c73638e29e0e6"}
Nov 03 13:35:55.193981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:55.193892    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" event={"ID":"52845948-0621-4d3a-895d-b57495762003","Type":"ContainerStarted","Data":"1800b7a4f282949b10a87dd02577ebbaaf8f504f0ed5ec47a61116b86abd8d01"}
Nov 03 13:35:55.224119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:55.224072    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" podStartSLOduration=2.224058935 podStartE2EDuration="2.224058935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:55.219849303 +0000 UTC m=+1596.297614329" watchObservedRunningTime="2026-11-03 13:35:55.224058935 +0000 UTC m=+1596.301823961"
Nov 03 13:35:55.253149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:55.253112    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"]
Nov 03 13:35:55.258009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:55.257972    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf47qzft4"]
Nov 03 13:35:55.321432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:55.321397    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1c11b1a-1635-474c-bbdf-0860aeddf84b-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:55.959899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:55.959867    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:56.113634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:56.113604    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:56.119740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:56.119716    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:56.201802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:56.201769    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:56.202813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:56.202794    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:35:56.786059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:56.784247    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:57.081914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.081801    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8" podUID="41763fbe-38f6-4519-becd-ce0e6f7c5110" containerName="payload-processing" containerID="cri-o://89fb0bc470cf7b4e099890ebd06f7bb85bb11e223679c4a3128428fd3d0ae07f" gracePeriod=25
Nov 03 13:35:57.086448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.086424    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c11b1a-1635-474c-bbdf-0860aeddf84b" path="/var/lib/kubelet/pods/d1c11b1a-1635-474c-bbdf-0860aeddf84b/volumes"
Nov 03 13:35:57.090587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.090561    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs" podUID="69480ba8-e96f-447d-af1e-2d78ab7116c5" containerName="payload-processing" containerID="cri-o://fa9d8f801b8bf27e5725d876ea5fca5e4a13df7039788379b76bd2787567253e" gracePeriod=25
Nov 03 13:35:57.221132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.221104    5420 generic.go:356] "Generic (PLEG): container finished" podID="41763fbe-38f6-4519-becd-ce0e6f7c5110" containerID="89fb0bc470cf7b4e099890ebd06f7bb85bb11e223679c4a3128428fd3d0ae07f" exitCode=0
Nov 03 13:35:57.221369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.221249    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8" event={"ID":"41763fbe-38f6-4519-becd-ce0e6f7c5110","Type":"ContainerDied","Data":"89fb0bc470cf7b4e099890ebd06f7bb85bb11e223679c4a3128428fd3d0ae07f"}
Nov 03 13:35:57.223185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.223162    5420 generic.go:356] "Generic (PLEG): container finished" podID="69480ba8-e96f-447d-af1e-2d78ab7116c5" containerID="fa9d8f801b8bf27e5725d876ea5fca5e4a13df7039788379b76bd2787567253e" exitCode=0
Nov 03 13:35:57.223315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.223238    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs" event={"ID":"69480ba8-e96f-447d-af1e-2d78ab7116c5","Type":"ContainerDied","Data":"fa9d8f801b8bf27e5725d876ea5fca5e4a13df7039788379b76bd2787567253e"}
Nov 03 13:35:57.399561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.399524    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:57.411881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.411848    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:57.460105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.460071    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:57.460463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.460442    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-kube-api-access-9g9r6\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-kube-api-access-9g9r6\") pod \"69480ba8-e96f-447d-af1e-2d78ab7116c5\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") "
Nov 03 13:35:57.460566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.460540    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-plugins-config-volume\") pod \"41763fbe-38f6-4519-becd-ce0e6f7c5110\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") "
Nov 03 13:35:57.460673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.460658    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-provider-credentials\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-provider-credentials\") pod \"41763fbe-38f6-4519-becd-ce0e6f7c5110\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") "
Nov 03 13:35:57.460745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.460697    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-routing-overlay-volume\") pod \"41763fbe-38f6-4519-becd-ce0e6f7c5110\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") "
Nov 03 13:35:57.460802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.460762    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-plugins-config-volume\") pod \"69480ba8-e96f-447d-af1e-2d78ab7116c5\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") "
Nov 03 13:35:57.460865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.460805    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-provider-credentials\") pod \"69480ba8-e96f-447d-af1e-2d78ab7116c5\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") "
Nov 03 13:35:57.461762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.461735    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-routing-overlay-volume" pod "41763fbe-38f6-4519-becd-ce0e6f7c5110" (UID: "41763fbe-38f6-4519-becd-ce0e6f7c5110"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:57.462199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.462175    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-plugins-config-volume" pod "69480ba8-e96f-447d-af1e-2d78ab7116c5" (UID: "69480ba8-e96f-447d-af1e-2d78ab7116c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:57.462998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.462970    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-plugins-config-volume" pod "41763fbe-38f6-4519-becd-ce0e6f7c5110" (UID: "41763fbe-38f6-4519-becd-ce0e6f7c5110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:57.463175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.463073    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-routing-overlay-volume\") pod \"69480ba8-e96f-447d-af1e-2d78ab7116c5\" (UID: \"69480ba8-e96f-447d-af1e-2d78ab7116c5\") "
Nov 03 13:35:57.463311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.463253    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-kube-api-access-vmvgg\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-kube-api-access-vmvgg\") pod \"41763fbe-38f6-4519-becd-ce0e6f7c5110\" (UID: \"41763fbe-38f6-4519-becd-ce0e6f7c5110\") "
Nov 03 13:35:57.463825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.463801    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-provider-credentials" pod "41763fbe-38f6-4519-becd-ce0e6f7c5110" (UID: "41763fbe-38f6-4519-becd-ce0e6f7c5110"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:57.464051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.464028    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-kube-api-access-9g9r6" pod "69480ba8-e96f-447d-af1e-2d78ab7116c5" (UID: "69480ba8-e96f-447d-af1e-2d78ab7116c5"). InnerVolumeSpecName "kube-api-access-9g9r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:57.464416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.464395    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-routing-overlay-volume" pod "69480ba8-e96f-447d-af1e-2d78ab7116c5" (UID: "69480ba8-e96f-447d-af1e-2d78ab7116c5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:35:57.464942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.464902    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:57.464942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.464929    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:57.465051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.464944    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:57.465051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.464959    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69480ba8-e96f-447d-af1e-2d78ab7116c5-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:57.465051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.464972    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9r6\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-kube-api-access-9g9r6\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:57.465051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.464988    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41763fbe-38f6-4519-becd-ce0e6f7c5110-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:57.465307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.465285    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-provider-credentials" pod "69480ba8-e96f-447d-af1e-2d78ab7116c5" (UID: "69480ba8-e96f-447d-af1e-2d78ab7116c5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:57.465792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.465773    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-kube-api-access-vmvgg" pod "41763fbe-38f6-4519-becd-ce0e6f7c5110" (UID: "41763fbe-38f6-4519-becd-ce0e6f7c5110"). InnerVolumeSpecName "kube-api-access-vmvgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:35:57.566827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.566792    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmvgg\" (UniqueName: \"kubernetes.io/projected/41763fbe-38f6-4519-becd-ce0e6f7c5110-kube-api-access-vmvgg\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:57.566827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.566820    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69480ba8-e96f-447d-af1e-2d78ab7116c5-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:35:57.626296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:57.624832    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:58.233199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.233166    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"
Nov 03 13:35:58.233684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.233205    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8" event={"ID":"41763fbe-38f6-4519-becd-ce0e6f7c5110","Type":"ContainerDied","Data":"e6f8a152c1669a36246b811f8de1038a8cb60907fea5b84b4fa82c15069eaded"}
Nov 03 13:35:58.233684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.233311    5420 scope.go:122] "RemoveContainer" containerID="89fb0bc470cf7b4e099890ebd06f7bb85bb11e223679c4a3128428fd3d0ae07f"
Nov 03 13:35:58.235201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.235179    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"
Nov 03 13:35:58.235372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.235287    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs" event={"ID":"69480ba8-e96f-447d-af1e-2d78ab7116c5","Type":"ContainerDied","Data":"1f0c20bc4c4ddd724771e9eadf73cc5d88caeed74d862d9ae1dfbc72655ef271"}
Nov 03 13:35:58.247138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.247119    5420 scope.go:122] "RemoveContainer" containerID="fa9d8f801b8bf27e5725d876ea5fca5e4a13df7039788379b76bd2787567253e"
Nov 03 13:35:58.262783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.262756    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"]
Nov 03 13:35:58.267744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.267718    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffjv8m8"]
Nov 03 13:35:58.280859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.280822    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"]
Nov 03 13:35:58.285499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.285473    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs"]
Nov 03 13:35:58.373910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.373810    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-57dd42rfs" podUID="69480ba8-e96f-447d-af1e-2d78ab7116c5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.212:50052\" within 1s: context deadline exceeded"
Nov 03 13:35:58.374140 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:35:58.373933    5420 logging.go:55] [core] [Channel #4565 SubChannel #4566]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.212:50052", ServerName: "10.134.0.212:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.212:50052: operation was canceled"
Nov 03 13:35:58.410938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.410895    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:58.777490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.777451    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:58.816814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.816780    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"]
Nov 03 13:35:58.817528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.817512    5420 cpu_manager.go:414] "containerMap: removing container" podUID="69480ba8-e96f-447d-af1e-2d78ab7116c5" containerName="payload-processing"
Nov 03 13:35:58.817614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.817529    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69480ba8-e96f-447d-af1e-2d78ab7116c5" containerName="payload-processing"
Nov 03 13:35:58.817614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.817545    5420 cpu_manager.go:414] "containerMap: removing container" podUID="41763fbe-38f6-4519-becd-ce0e6f7c5110" containerName="payload-processing"
Nov 03 13:35:58.817614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.817551    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41763fbe-38f6-4519-becd-ce0e6f7c5110" containerName="payload-processing"
Nov 03 13:35:58.817773 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.817700    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="69480ba8-e96f-447d-af1e-2d78ab7116c5" containerName="payload-processing"
Nov 03 13:35:58.817773 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.817709    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="41763fbe-38f6-4519-becd-ce0e6f7c5110" containerName="payload-processing"
Nov 03 13:35:58.823288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.823265    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:35:58.829553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.829528    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-9f71f2-dockercfg-5wn4s\""
Nov 03 13:35:58.830148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.830121    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-9f71f2\""
Nov 03 13:35:58.832328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.832307    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"]
Nov 03 13:35:58.906240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.899555    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5nqh\" (UniqueName: \"kubernetes.io/projected/8285a1b2-2969-4bdc-8385-aede1a87dbb6-kube-api-access-j5nqh\") pod \"payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl\" (UID: \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:35:58.906240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.899935    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285a1b2-2969-4bdc-8385-aede1a87dbb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl\" (UID: \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:35:58.980245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:58.977746    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:59.006235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:59.003607    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285a1b2-2969-4bdc-8385-aede1a87dbb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl\" (UID: \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:35:59.006235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:59.003879    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5nqh\" (UniqueName: \"kubernetes.io/projected/8285a1b2-2969-4bdc-8385-aede1a87dbb6-kube-api-access-j5nqh\") pod \"payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl\" (UID: \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:35:59.006235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:59.005261    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285a1b2-2969-4bdc-8385-aede1a87dbb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl\" (UID: \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:35:59.025237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:59.025012    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5nqh\" (UniqueName: \"kubernetes.io/projected/8285a1b2-2969-4bdc-8385-aede1a87dbb6-kube-api-access-j5nqh\") pod \"payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl\" (UID: \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:35:59.107320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:59.105372    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41763fbe-38f6-4519-becd-ce0e6f7c5110" path="/var/lib/kubelet/pods/41763fbe-38f6-4519-becd-ce0e6f7c5110/volumes"
Nov 03 13:35:59.107320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:59.105914    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69480ba8-e96f-447d-af1e-2d78ab7116c5" path="/var/lib/kubelet/pods/69480ba8-e96f-447d-af1e-2d78ab7116c5/volumes"
Nov 03 13:35:59.149697 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:59.149553    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:35:59.421292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:59.421169    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:59.470734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:35:59.470707    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"]
Nov 03 13:36:00.255329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:00.251693    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:00.278236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:00.266899    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl" event={"ID":"8285a1b2-2969-4bdc-8385-aede1a87dbb6","Type":"ContainerStarted","Data":"eb46fdd87d879647b87c3a3d1391d3d982dce124c52424e2a4522ad894ea5229"}
Nov 03 13:36:00.278236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:00.266954    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl" event={"ID":"8285a1b2-2969-4bdc-8385-aede1a87dbb6","Type":"ContainerStarted","Data":"d566f711c76e0adbba2da0a8f0c6de050edb2640e680cf451d4adb27934af3d4"}
Nov 03 13:36:00.278236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:00.267939    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:36:00.301711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:00.299547    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl" podStartSLOduration=2.299525534 podStartE2EDuration="2.299525534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:00.294568833 +0000 UTC m=+1601.372333859" watchObservedRunningTime="2026-11-03 13:36:00.299525534 +0000 UTC m=+1601.377290563"
Nov 03 13:36:00.509440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:00.509359    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:00.690798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:00.689391    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:01.181288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:01.173305    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:02.046905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:02.046875    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:02.403981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:02.403893    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:02.592891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:02.592829    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:02.954772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:02.954742    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:03.145339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:03.145311    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:36:03.145855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:03.145675    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:36:04.113791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:04.113758    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:04.266690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:04.266658    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:04.940805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:04.940770    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:05.531407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:05.531308    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:06.139877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:06.139845    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:06.808612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:06.808579    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:07.565266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:07.563597    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:08.225609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:08.225574    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:09.059140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:09.059101    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:09.243897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:09.243858    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:09.419189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:09.419094    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:09.840169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:09.840116    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:10.012502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.012468    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:10.223888 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.223853    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"]
Nov 03 13:36:10.229775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.229752    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.247034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.247010    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"]
Nov 03 13:36:10.315740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.315708    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-146318-openshift-default-dockercfg-dlt2v\""
Nov 03 13:36:10.336310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.321224    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-socket\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.336310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.321366    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-token\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.336310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.321406    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-envoy\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.336310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.321432    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2fs\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-kube-api-access-xx2fs\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.336310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.321537    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-certs\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.336310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.321600    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-credential-socket\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.336310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.321628    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istiod-ca-cert\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.336310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.321675    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-data\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.336310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.321704    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-podinfo\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.378858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.378529    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:10.423144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.423110    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-token\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.423407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.423162    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-envoy\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.423407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.423185    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2fs\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-kube-api-access-xx2fs\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.423407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.423292    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-certs\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.423407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.423355    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-credential-socket\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.423407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.423378    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istiod-ca-cert\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.423675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.423420    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-data\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.423675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.423447    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-podinfo\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.423675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.423620    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-socket\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.425613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.425582    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-socket\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.426153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.426130    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-certs\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.426479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.426454    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-data\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.426682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.426662    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-credential-socket\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.427098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.427071    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istiod-ca-cert\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.429372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.429353    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-envoy\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.430890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.430867    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-podinfo\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.441948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.441915    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-token\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.442141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.442045    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2fs\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-kube-api-access-xx2fs\") pod \"e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") " pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.526893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.526800    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:10.551051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.551018    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:10.726276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.726195    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:10.886273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.881463    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"]
Nov 03 13:36:10.886273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.886169    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:36:10.886273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.886265    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:36:10.886779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:10.886315    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:36:11.395954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:11.395923    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" event={"ID":"4c550cb4-9c69-4cf7-a9c8-560011a88b56","Type":"ContainerStarted","Data":"c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44"}
Nov 03 13:36:11.396420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:11.395974    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" event={"ID":"4c550cb4-9c69-4cf7-a9c8-560011a88b56","Type":"ContainerStarted","Data":"2c953ff97b7aac59766e6609c82b05e16232512e7ecaf5953612345b46ff0fae"}
Nov 03 13:36:11.428914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:11.428864    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" podStartSLOduration=1.428848457 podStartE2EDuration="1.428848457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:11.424444545 +0000 UTC m=+1612.502209609" watchObservedRunningTime="2026-11-03 13:36:11.428848457 +0000 UTC m=+1612.506613482"
Nov 03 13:36:11.553651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:11.551538    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:11.570234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:11.565516    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:11.575741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:11.575458    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:12.161411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:12.161380    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:12.301662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:12.301635    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:36:12.307343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:12.307305    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:12.404907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:12.404879    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:12.405821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:12.405792    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:36:13.368024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:13.367971    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:13.807946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:13.807904    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:13.976177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:13.976150    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:14.266030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:14.265987    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:14.928878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:14.928834    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:15.100481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.100455    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:15.154263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.154234    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"]
Nov 03 13:36:15.159465 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.159442    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:15.166392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.166364    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-146318\""
Nov 03 13:36:15.167094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.167075    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-146318-dockercfg-p5js4\""
Nov 03 13:36:15.167981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.167963    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"]
Nov 03 13:36:15.238560 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.238531    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pncnx\" (UniqueName: \"kubernetes.io/projected/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-kube-api-access-pncnx\") pod \"payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr\" (UID: \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:15.238738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.238720    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr\" (UID: \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:15.257770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.257729    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:15.343233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.342718    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr\" (UID: \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:15.343233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.343007    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pncnx\" (UniqueName: \"kubernetes.io/projected/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-kube-api-access-pncnx\") pod \"payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr\" (UID: \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:15.518319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.518063    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr\" (UID: \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:15.531006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.530970    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pncnx\" (UniqueName: \"kubernetes.io/projected/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-kube-api-access-pncnx\") pod \"payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr\" (UID: \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:15.540829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.540796    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"]
Nov 03 13:36:15.556277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.548619    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:15.556277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.552525    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"]
Nov 03 13:36:15.662727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.662688    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqhl\" (UniqueName: \"kubernetes.io/projected/482e5ea4-ff10-4559-8299-4e0bf09c39d0-kube-api-access-gjqhl\") pod \"payload-processing-e2e-autores-146318-7c785754ff-s472g\" (UID: \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\") " pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:15.663053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.663017    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482e5ea4-ff10-4559-8299-4e0bf09c39d0-plugins-config-volume\") pod \"payload-processing-e2e-autores-146318-7c785754ff-s472g\" (UID: \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\") " pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:15.718297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.718266    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:15.764145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.764110    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqhl\" (UniqueName: \"kubernetes.io/projected/482e5ea4-ff10-4559-8299-4e0bf09c39d0-kube-api-access-gjqhl\") pod \"payload-processing-e2e-autores-146318-7c785754ff-s472g\" (UID: \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\") " pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:15.764389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.764349    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482e5ea4-ff10-4559-8299-4e0bf09c39d0-plugins-config-volume\") pod \"payload-processing-e2e-autores-146318-7c785754ff-s472g\" (UID: \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\") " pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:15.765051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.765032    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482e5ea4-ff10-4559-8299-4e0bf09c39d0-plugins-config-volume\") pod \"payload-processing-e2e-autores-146318-7c785754ff-s472g\" (UID: \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\") " pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:15.779742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.779682    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:15.780755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.780733    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqhl\" (UniqueName: \"kubernetes.io/projected/482e5ea4-ff10-4559-8299-4e0bf09c39d0-kube-api-access-gjqhl\") pod \"payload-processing-e2e-autores-146318-7c785754ff-s472g\" (UID: \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\") " pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:15.892048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:15.891463    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:16.100553 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:36:16.100524    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43ea246_19cd_4466_bc4e_f7eecb3b22fc.slice/crio-1d276c453e6bd47d3be1cabf8fdb34bb7997ba613dec3b6548b82c6b78bc96e0 WatchSource:0}: Error finding container 1d276c453e6bd47d3be1cabf8fdb34bb7997ba613dec3b6548b82c6b78bc96e0: Status 404 returned error can't find the container with id 1d276c453e6bd47d3be1cabf8fdb34bb7997ba613dec3b6548b82c6b78bc96e0
Nov 03 13:36:16.101556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.101516    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"]
Nov 03 13:36:16.208647 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:36:16.208619    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482e5ea4_ff10_4559_8299_4e0bf09c39d0.slice/crio-a5d64dc2ff803afa8db1d2a17db106365b8a48d26a5e9e616e1aa40d667ef6ce WatchSource:0}: Error finding container a5d64dc2ff803afa8db1d2a17db106365b8a48d26a5e9e616e1aa40d667ef6ce: Status 404 returned error can't find the container with id a5d64dc2ff803afa8db1d2a17db106365b8a48d26a5e9e616e1aa40d667ef6ce
Nov 03 13:36:16.209753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.209717    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"]
Nov 03 13:36:16.466755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.466707    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:16.525103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.525069    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr" event={"ID":"f43ea246-19cd-4466-bc4e-f7eecb3b22fc","Type":"ContainerStarted","Data":"5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7"}
Nov 03 13:36:16.525370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.525133    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr" event={"ID":"f43ea246-19cd-4466-bc4e-f7eecb3b22fc","Type":"ContainerStarted","Data":"1d276c453e6bd47d3be1cabf8fdb34bb7997ba613dec3b6548b82c6b78bc96e0"}
Nov 03 13:36:16.525370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.525294    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:16.527160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.527132    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g" event={"ID":"482e5ea4-ff10-4559-8299-4e0bf09c39d0","Type":"ContainerStarted","Data":"5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10"}
Nov 03 13:36:16.527279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.527182    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g" event={"ID":"482e5ea4-ff10-4559-8299-4e0bf09c39d0","Type":"ContainerStarted","Data":"a5d64dc2ff803afa8db1d2a17db106365b8a48d26a5e9e616e1aa40d667ef6ce"}
Nov 03 13:36:16.527279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.527206    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:16.566585 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.566536    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr" podStartSLOduration=1.5665220930000001 podStartE2EDuration="1.566522093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:16.546949913 +0000 UTC m=+1617.624714938" watchObservedRunningTime="2026-11-03 13:36:16.566522093 +0000 UTC m=+1617.644287160"
Nov 03 13:36:16.568866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:16.568819    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g" podStartSLOduration=1.56880488 podStartE2EDuration="1.56880488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:16.564933848 +0000 UTC m=+1617.642698875" watchObservedRunningTime="2026-11-03 13:36:16.56880488 +0000 UTC m=+1617.646569907"
Nov 03 13:36:17.280871 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:17.280829    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:17.929240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:17.928824    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:18.026592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:18.026558    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"]
Nov 03 13:36:18.031191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:18.031170    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004aec0b-7hls6"]
Nov 03 13:36:18.496251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:18.496064    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:18.779862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:18.779753    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:19.101937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:19.101849    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:19.108855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:19.108827    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd828ff-07e8-49e8-8359-73395f78b6db" path="/var/lib/kubelet/pods/0dd828ff-07e8-49e8-8359-73395f78b6db/volumes"
Nov 03 13:36:19.289890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:19.289861    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:19.808372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:19.808331    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:20.690269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:20.680438    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:21.328265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:21.327322    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:21.921440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:21.921406    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:22.528284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:22.528145    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:23.302337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:23.302302    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:23.860843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:23.860808    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:24.029654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:24.029621    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:24.845236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:24.844118    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:24.881403 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:36:24.881370    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:36:25.055644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:25.055615    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:25.437658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:25.437625    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:25.636450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:25.636412    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:26.356864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:26.356706    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:26.645262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:26.645155    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:27.224116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:27.224077    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:27.315734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:27.315684    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"]
Nov 03 13:36:27.456622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:27.456591    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:27.538766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:27.538682    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:36:27.538948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:27.538916    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:36:27.599476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:27.599424    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:28.027815 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:28.027783    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"]
Nov 03 13:36:28.032924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:28.032896    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75212753-hv52l"]
Nov 03 13:36:28.200720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:28.200692    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:28.343164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:28.343064    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:28.571127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:28.571094    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:29.014203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:29.014169    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:29.088621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:29.088589    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1420976-939e-4a5a-96db-c41d7c234201" path="/var/lib/kubelet/pods/b1420976-939e-4a5a-96db-c41d7c234201/volumes"
Nov 03 13:36:29.265477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:29.265393    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:36:29.904993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:29.904960    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:30.864244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:30.864159    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:31.884128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:31.884084    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:32.316002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.315967    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8" podUID="e94a1950-4189-4de8-9a4d-5a34ed885087" containerName="payload-processing" containerID="cri-o://71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af" gracePeriod=25
Nov 03 13:36:32.662425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.662399    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:36:32.756537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.756510    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94a1950-4189-4de8-9a4d-5a34ed885087-kube-api-access-nkfpk\" (UniqueName: \"kubernetes.io/projected/e94a1950-4189-4de8-9a4d-5a34ed885087-kube-api-access-nkfpk\") pod \"e94a1950-4189-4de8-9a4d-5a34ed885087\" (UID: \"e94a1950-4189-4de8-9a4d-5a34ed885087\") "
Nov 03 13:36:32.756879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.756858    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94a1950-4189-4de8-9a4d-5a34ed885087-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94a1950-4189-4de8-9a4d-5a34ed885087-plugins-config-volume\") pod \"e94a1950-4189-4de8-9a4d-5a34ed885087\" (UID: \"e94a1950-4189-4de8-9a4d-5a34ed885087\") "
Nov 03 13:36:32.757992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.757966    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94a1950-4189-4de8-9a4d-5a34ed885087-plugins-config-volume" pod "e94a1950-4189-4de8-9a4d-5a34ed885087" (UID: "e94a1950-4189-4de8-9a4d-5a34ed885087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:36:32.760282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.760252    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94a1950-4189-4de8-9a4d-5a34ed885087-kube-api-access-nkfpk" pod "e94a1950-4189-4de8-9a4d-5a34ed885087" (UID: "e94a1950-4189-4de8-9a4d-5a34ed885087"). InnerVolumeSpecName "kube-api-access-nkfpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:32.761965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.761937    5420 generic.go:356] "Generic (PLEG): container finished" podID="e94a1950-4189-4de8-9a4d-5a34ed885087" containerID="71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af" exitCode=0
Nov 03 13:36:32.762265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.762002    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"
Nov 03 13:36:32.763040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.762030    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8" event={"ID":"e94a1950-4189-4de8-9a4d-5a34ed885087","Type":"ContainerDied","Data":"71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af"}
Nov 03 13:36:32.763140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.763066    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8" event={"ID":"e94a1950-4189-4de8-9a4d-5a34ed885087","Type":"ContainerDied","Data":"6ca20c11c5ab9785d68389c60a307f0031c706ec90fc99597f54a61aa452d6a0"}
Nov 03 13:36:32.763140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.763101    5420 scope.go:122] "RemoveContainer" containerID="71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af"
Nov 03 13:36:32.796019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.795986    5420 scope.go:122] "RemoveContainer" containerID="71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af"
Nov 03 13:36:32.796452 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:36:32.796411    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af\": container with ID starting with 71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af not found: ID does not exist" containerID="71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af"
Nov 03 13:36:32.796515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.796464    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af"} err="failed to get container status \"71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af\": rpc error: code = NotFound desc = could not find container \"71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af\": container with ID starting with 71d564ba01e9df066fee812d36050715622e50af5b87439f55cf5f7560b636af not found: ID does not exist"
Nov 03 13:36:32.806039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.806017    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"]
Nov 03 13:36:32.820195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.820174    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1ec4b892-648fc646cf-wb6p8"]
Nov 03 13:36:32.859484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.859383    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94a1950-4189-4de8-9a4d-5a34ed885087-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:32.859484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.859415    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkfpk\" (UniqueName: \"kubernetes.io/projected/e94a1950-4189-4de8-9a4d-5a34ed885087-kube-api-access-nkfpk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:32.963974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.963941    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"]
Nov 03 13:36:32.964802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.964783    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e94a1950-4189-4de8-9a4d-5a34ed885087" containerName="payload-processing"
Nov 03 13:36:32.964802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.964803    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94a1950-4189-4de8-9a4d-5a34ed885087" containerName="payload-processing"
Nov 03 13:36:32.964967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.964956    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94a1950-4189-4de8-9a4d-5a34ed885087" containerName="payload-processing"
Nov 03 13:36:32.970618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.970593    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:32.979709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.976859    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a724a6-openshift-default-dockercfg-l5hxx\""
Nov 03 13:36:32.982373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.982350    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:32.982940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:32.982918    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"]
Nov 03 13:36:33.107707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.107670    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94a1950-4189-4de8-9a4d-5a34ed885087" path="/var/lib/kubelet/pods/e94a1950-4189-4de8-9a4d-5a34ed885087/volumes"
Nov 03 13:36:33.151159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.151067    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:33.189474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.189443    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.189617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.189566    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.189674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.189626    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.189731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.189715    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.189869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.189851    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwv6\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-kube-api-access-6bwv6\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.189910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.189893    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.189946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.189937    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.190008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.189996    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.190045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.190015    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291200    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwv6\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-kube-api-access-6bwv6\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291258    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291298    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291333    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291349    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291393    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291461    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291524    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291595    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291750    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291778    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291835    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.291896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.291843    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.292297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.292275    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.293834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.293804    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.294150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.294130    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.299650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.299628    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwv6\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-kube-api-access-6bwv6\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.299757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.299709    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.590892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.590856    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:33.898220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.897940    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:33.919170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.919145    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"]
Nov 03 13:36:33.927229 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:36:33.925500    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b831a0b_a2a2_4fcc_8cd6_74d45b48dfab.slice/crio-a734950490c13e17a11d76559a53c9517645aee4143bd888ce6ee20763d5a024 WatchSource:0}: Error finding container a734950490c13e17a11d76559a53c9517645aee4143bd888ce6ee20763d5a024: Status 404 returned error can't find the container with id a734950490c13e17a11d76559a53c9517645aee4143bd888ce6ee20763d5a024
Nov 03 13:36:33.955232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.953729    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:36:33.955232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.953868    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:36:33.955232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:33.953916    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:36:34.127821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:34.127788    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:34.400590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:34.400550    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:34.566236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:34.565788    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:34.818304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:34.818268    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:34.906851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:34.906818    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" event={"ID":"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab","Type":"ContainerStarted","Data":"3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88"}
Nov 03 13:36:34.907129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:34.907110    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" event={"ID":"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab","Type":"ContainerStarted","Data":"a734950490c13e17a11d76559a53c9517645aee4143bd888ce6ee20763d5a024"}
Nov 03 13:36:34.970078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:34.963954    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" podStartSLOduration=2.963932566 podStartE2EDuration="2.963932566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:34.938367193 +0000 UTC m=+1636.016132221" watchObservedRunningTime="2026-11-03 13:36:34.963932566 +0000 UTC m=+1636.041697593"
Nov 03 13:36:35.456382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:35.456343    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:35.602234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:35.598908    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:35.609557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:35.604929    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:36:35.609557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:35.605783    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:35.609557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:35.605862    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 13:36:36.202824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:36.202784    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:36.592103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:36.592024    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:36:36.592103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:36.592074    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 13:36:36.688781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:36.688725    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:37.206901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:37.206871    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:37.618273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:37.618163    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:36:37.619448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:37.619409    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 13:36:37.620513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:37.620486    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:38.354232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:38.352101    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:38.592203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:38.592168    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:36:38.592411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:38.592258    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 13:36:39.142234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:39.142160    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:39.591407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:39.591376    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:36:39.591577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:39.591427    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 13:36:39.717251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:39.717124    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:40.345357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:40.345108    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:40.591863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:40.591825    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 13:36:40.592043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:40.591896    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 13:36:41.017657 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:41.017615    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:41.597072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:41.597044    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:41.598221 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:41.598190    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:41.598856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:41.598835    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:36:41.646073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:41.645974    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:41.967921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:41.967879    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:42.148105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:42.148077    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:42.243404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:42.243326    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:36:42.243404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:42.243380    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:36:42.518029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:42.517921    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:42.692162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:42.692070    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:42.879346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:42.879269    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"]
Nov 03 13:36:42.879539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:42.879505    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy" containerID="cri-o://76ea23fc100e66da057b5095cb6b23463d0abb5e03b911d0f1ae09ec42c1e8cb" gracePeriod=30
Nov 03 13:36:42.974232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:42.970401    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:43.122910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:43.116437    5420 generic.go:356] "Generic (PLEG): container finished" podID="52845948-0621-4d3a-895d-b57495762003" containerID="2888860214ac067a034256b03f867aed3bf06663f415604f7c3c73638e29e0e6" exitCode=0
Nov 03 13:36:43.135311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:43.133317    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" event={"ID":"52845948-0621-4d3a-895d-b57495762003","Type":"ContainerDied","Data":"2888860214ac067a034256b03f867aed3bf06663f415604f7c3c73638e29e0e6"}
Nov 03 13:36:43.136455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:43.136429    5420 scope.go:122] "RemoveContainer" containerID="2888860214ac067a034256b03f867aed3bf06663f415604f7c3c73638e29e0e6"
Nov 03 13:36:43.140531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:43.140496    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:36:43.140657 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:43.140605    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:36:43.140722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:43.140655    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:36:43.409642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:43.409561    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:44.036234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.034361    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:44.045889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.045857    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"]
Nov 03 13:36:44.050797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.050769    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf68801-92m8l"]
Nov 03 13:36:44.152236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.152066    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" event={"ID":"52845948-0621-4d3a-895d-b57495762003","Type":"ContainerStarted","Data":"59ed8135f6896468e3eb0485ce30397bd3de5967e4d8cb15f872ba71e3ecc4c3"}
Nov 03 13:36:44.355260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.354397    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"]
Nov 03 13:36:44.360341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.360315    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"]
Nov 03 13:36:44.360544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.360370    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.371586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.371559    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:36:44.371766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.371754    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:36:44.371829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.371819    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:36:44.371873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.371860    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:36:44.458317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.458278    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3729b590-vgb7v\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.458540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.458417    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3729b590-vgb7v\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.458617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.458553    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qln5j\" (UniqueName: \"kubernetes.io/projected/7e0b1f42-4664-4408-89c6-056f7458788e-kube-api-access-qln5j\") pod \"maas-api-revoke-sub-3729b590-vgb7v\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.559776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.559737    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qln5j\" (UniqueName: \"kubernetes.io/projected/7e0b1f42-4664-4408-89c6-056f7458788e-kube-api-access-qln5j\") pod \"maas-api-revoke-sub-3729b590-vgb7v\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.560077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.560058    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3729b590-vgb7v\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.560171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.560141    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3729b590-vgb7v\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.560716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.560692    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3729b590-vgb7v\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.562515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.562492    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3729b590-vgb7v\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.569116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.569098    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qln5j\" (UniqueName: \"kubernetes.io/projected/7e0b1f42-4664-4408-89c6-056f7458788e-kube-api-access-qln5j\") pod \"maas-api-revoke-sub-3729b590-vgb7v\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.678489 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.678411    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:44.888766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.888335    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:44.997609 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:36:44.997572    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b1f42_4664_4408_89c6_056f7458788e.slice/crio-16c5edc395f4744dee285f6ea9f9b8cc277887b3c2b46a1868cde2931f8af863 WatchSource:0}: Error finding container 16c5edc395f4744dee285f6ea9f9b8cc277887b3c2b46a1868cde2931f8af863: Status 404 returned error can't find the container with id 16c5edc395f4744dee285f6ea9f9b8cc277887b3c2b46a1868cde2931f8af863
Nov 03 13:36:44.998864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:44.998834    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"]
Nov 03 13:36:45.086587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:45.086551    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2940d67-1ee4-40af-ad03-195e9da36635" path="/var/lib/kubelet/pods/b2940d67-1ee4-40af-ad03-195e9da36635/volumes"
Nov 03 13:36:45.113640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:45.113615    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:36:45.113836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:45.113814    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 13:36:45.113887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:45.113867    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 13:36:45.162119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:45.162088    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v" event={"ID":"7e0b1f42-4664-4408-89c6-056f7458788e","Type":"ContainerStarted","Data":"16c5edc395f4744dee285f6ea9f9b8cc277887b3c2b46a1868cde2931f8af863"}
Nov 03 13:36:45.598367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:45.594401    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:46.114288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:46.114256    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 13:36:46.114703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:46.114309    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 13:36:46.174024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:46.173985    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v" event={"ID":"7e0b1f42-4664-4408-89c6-056f7458788e","Type":"ContainerStarted","Data":"81617a2888968ca00f801d56d6a8701d7ca2dc4c694d73f3bb3f8070fe40b70f"}
Nov 03 13:36:46.268346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:46.266143    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:46.902236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:46.895786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:47.114047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.114007    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 13:36:47.114206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.114063    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 13:36:47.184452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.184360    5420 generic.go:356] "Generic (PLEG): container finished" podID="7e0b1f42-4664-4408-89c6-056f7458788e" containerID="81617a2888968ca00f801d56d6a8701d7ca2dc4c694d73f3bb3f8070fe40b70f" exitCode=0
Nov 03 13:36:47.184452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.184423    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v" event={"ID":"7e0b1f42-4664-4408-89c6-056f7458788e","Type":"ContainerDied","Data":"81617a2888968ca00f801d56d6a8701d7ca2dc4c694d73f3bb3f8070fe40b70f"}
Nov 03 13:36:47.407388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.407362    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:47.480745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.480711    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-openshift-ca\") pod \"7e0b1f42-4664-4408-89c6-056f7458788e\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") "
Nov 03 13:36:47.480989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.480805    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0b1f42-4664-4408-89c6-056f7458788e-kube-api-access-qln5j\" (UniqueName: \"kubernetes.io/projected/7e0b1f42-4664-4408-89c6-056f7458788e-kube-api-access-qln5j\") pod \"7e0b1f42-4664-4408-89c6-056f7458788e\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") "
Nov 03 13:36:47.480989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.480867    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-xks-ca\") pod \"7e0b1f42-4664-4408-89c6-056f7458788e\" (UID: \"7e0b1f42-4664-4408-89c6-056f7458788e\") "
Nov 03 13:36:47.481204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.481177    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-openshift-ca" pod "7e0b1f42-4664-4408-89c6-056f7458788e" (UID: "7e0b1f42-4664-4408-89c6-056f7458788e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:36:47.481643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.481619    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:47.485252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.483015    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-xks-ca" pod "7e0b1f42-4664-4408-89c6-056f7458788e" (UID: "7e0b1f42-4664-4408-89c6-056f7458788e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:36:47.485252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.483036    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0b1f42-4664-4408-89c6-056f7458788e-kube-api-access-qln5j" pod "7e0b1f42-4664-4408-89c6-056f7458788e" (UID: "7e0b1f42-4664-4408-89c6-056f7458788e"). InnerVolumeSpecName "kube-api-access-qln5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:47.569235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.568642    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:47.584381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.584339    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qln5j\" (UniqueName: \"kubernetes.io/projected/7e0b1f42-4664-4408-89c6-056f7458788e-kube-api-access-qln5j\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:47.584817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:47.584793    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b1f42-4664-4408-89c6-056f7458788e-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.117155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.113923    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 13:36:48.117155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.113997    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 13:36:48.138257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.136068    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:48.199990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.199904    5420 generic.go:356] "Generic (PLEG): container finished" podID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerID="76ea23fc100e66da057b5095cb6b23463d0abb5e03b911d0f1ae09ec42c1e8cb" exitCode=0
Nov 03 13:36:48.200440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.200196    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" event={"ID":"cf5edc82-7131-4b51-b078-ca321b6d1e16","Type":"ContainerDied","Data":"76ea23fc100e66da057b5095cb6b23463d0abb5e03b911d0f1ae09ec42c1e8cb"}
Nov 03 13:36:48.202032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.202008    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"
Nov 03 13:36:48.202238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.202013    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v" event={"ID":"7e0b1f42-4664-4408-89c6-056f7458788e","Type":"ContainerDied","Data":"16c5edc395f4744dee285f6ea9f9b8cc277887b3c2b46a1868cde2931f8af863"}
Nov 03 13:36:48.202342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.202275    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c5edc395f4744dee285f6ea9f9b8cc277887b3c2b46a1868cde2931f8af863"
Nov 03 13:36:48.255169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.255146    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:36:48.298069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.298037    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-token\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-token\") pod \"cf5edc82-7131-4b51-b078-ca321b6d1e16\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") "
Nov 03 13:36:48.298255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.298174    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-podinfo\") pod \"cf5edc82-7131-4b51-b078-ca321b6d1e16\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") "
Nov 03 13:36:48.298346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.298279    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-kube-api-access-vf24j\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-kube-api-access-vf24j\") pod \"cf5edc82-7131-4b51-b078-ca321b6d1e16\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") "
Nov 03 13:36:48.298346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.298313    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-socket\") pod \"cf5edc82-7131-4b51-b078-ca321b6d1e16\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") "
Nov 03 13:36:48.298464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.298409    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-credential-socket\") pod \"cf5edc82-7131-4b51-b078-ca321b6d1e16\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") "
Nov 03 13:36:48.298464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.298452    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-envoy\") pod \"cf5edc82-7131-4b51-b078-ca321b6d1e16\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") "
Nov 03 13:36:48.298719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.298683    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-socket" pod "cf5edc82-7131-4b51-b078-ca321b6d1e16" (UID: "cf5edc82-7131-4b51-b078-ca321b6d1e16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.298767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.298706    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-credential-socket" pod "cf5edc82-7131-4b51-b078-ca321b6d1e16" (UID: "cf5edc82-7131-4b51-b078-ca321b6d1e16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.298926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.298905    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5edc82-7131-4b51-b078-ca321b6d1e16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5edc82-7131-4b51-b078-ca321b6d1e16-istiod-ca-cert\") pod \"cf5edc82-7131-4b51-b078-ca321b6d1e16\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") "
Nov 03 13:36:48.299032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.299015    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-data\") pod \"cf5edc82-7131-4b51-b078-ca321b6d1e16\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") "
Nov 03 13:36:48.299140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.299123    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-certs\") pod \"cf5edc82-7131-4b51-b078-ca321b6d1e16\" (UID: \"cf5edc82-7131-4b51-b078-ca321b6d1e16\") "
Nov 03 13:36:48.299389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.299365    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5edc82-7131-4b51-b078-ca321b6d1e16-istiod-ca-cert" pod "cf5edc82-7131-4b51-b078-ca321b6d1e16" (UID: "cf5edc82-7131-4b51-b078-ca321b6d1e16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:36:48.299515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.299487    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-certs" pod "cf5edc82-7131-4b51-b078-ca321b6d1e16" (UID: "cf5edc82-7131-4b51-b078-ca321b6d1e16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.299984 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.299958    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.300095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.299988    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5edc82-7131-4b51-b078-ca321b6d1e16-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.300095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.300004    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.300095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.300026    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.300404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.300110    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-data" pod "cf5edc82-7131-4b51-b078-ca321b6d1e16" (UID: "cf5edc82-7131-4b51-b078-ca321b6d1e16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.301495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.301471    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-kube-api-access-vf24j" pod "cf5edc82-7131-4b51-b078-ca321b6d1e16" (UID: "cf5edc82-7131-4b51-b078-ca321b6d1e16"). InnerVolumeSpecName "kube-api-access-vf24j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:48.302104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.302081    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-envoy" pod "cf5edc82-7131-4b51-b078-ca321b6d1e16" (UID: "cf5edc82-7131-4b51-b078-ca321b6d1e16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:36:48.306621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.306595    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-token" pod "cf5edc82-7131-4b51-b078-ca321b6d1e16" (UID: "cf5edc82-7131-4b51-b078-ca321b6d1e16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:36:48.307793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.307767    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-podinfo" pod "cf5edc82-7131-4b51-b078-ca321b6d1e16" (UID: "cf5edc82-7131-4b51-b078-ca321b6d1e16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:36:48.401133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.401091    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.401133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.401121    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.401133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.401134    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf24j\" (UniqueName: \"kubernetes.io/projected/cf5edc82-7131-4b51-b078-ca321b6d1e16-kube-api-access-vf24j\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.401453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.401149    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.401453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.401158    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5edc82-7131-4b51-b078-ca321b6d1e16-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:36:48.695896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:48.695861    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:49.113606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:49.113581    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 13:36:49.113748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:49.113626    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 13:36:49.212466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:49.212436    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"
Nov 03 13:36:49.212914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:49.212470    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9" event={"ID":"cf5edc82-7131-4b51-b078-ca321b6d1e16","Type":"ContainerDied","Data":"f82f836af664650f385fa03f88a50ccd60346f9f19450064f657e4a1b15ce87e"}
Nov 03 13:36:49.212914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:49.212533    5420 scope.go:122] "RemoveContainer" containerID="76ea23fc100e66da057b5095cb6b23463d0abb5e03b911d0f1ae09ec42c1e8cb"
Nov 03 13:36:49.249526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:49.249194    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:49.266945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:49.266922    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"]
Nov 03 13:36:49.279817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:49.279785    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-q88k9"]
Nov 03 13:36:49.402002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:49.401904    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:50.094051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:50.094016    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:50.123168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:50.123138    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:36:50.123512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:50.123487    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:36:50.124337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:50.124316    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:36:50.793082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:50.792895    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:51.086369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:51.086295    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" path="/var/lib/kubelet/pods/cf5edc82-7131-4b51-b078-ca321b6d1e16/volumes"
Nov 03 13:36:51.420954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:51.419940    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:52.191232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:52.188805    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:52.854601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:52.854568    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:53.006164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:53.006128    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:53.841423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:53.841375    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:54.709234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:54.707764    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:54.839833 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:36:54.839794    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:36:54.988893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:54.988814    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:55.038027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:55.037992    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"]
Nov 03 13:36:55.046199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:55.046162    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4ccb5d-wrwwc"]
Nov 03 13:36:55.085814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:55.085784    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75503be4-ea90-45be-9925-3632c0853539" path="/var/lib/kubelet/pods/75503be4-ea90-45be-9925-3632c0853539/volumes"
Nov 03 13:36:55.760247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:55.756842    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:55.917764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:55.917731    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:56.165391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:56.165301    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:56.746239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:56.745989    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:57.276600 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.276569    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"]
Nov 03 13:36:57.277350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.277333    5420 cpu_manager.go:414] "containerMap: removing container" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy"
Nov 03 13:36:57.277422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.277352    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy"
Nov 03 13:36:57.277422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.277372    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7e0b1f42-4664-4408-89c6-056f7458788e" containerName="revoke-keys"
Nov 03 13:36:57.277422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.277377    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0b1f42-4664-4408-89c6-056f7458788e" containerName="revoke-keys"
Nov 03 13:36:57.277519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.277495    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0b1f42-4664-4408-89c6-056f7458788e" containerName="revoke-keys"
Nov 03 13:36:57.277519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.277511    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5edc82-7131-4b51-b078-ca321b6d1e16" containerName="istio-proxy"
Nov 03 13:36:57.283445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.283420    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.287470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.287447    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:36:57.288295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.288272    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"]
Nov 03 13:36:57.288670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.288652    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:36:57.300360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.300336    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:36:57.300494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.300450    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:36:57.355744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.353994    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79r7w\" (UniqueName: \"kubernetes.io/projected/75736161-30da-4f0e-9e00-d5c951ea1161-kube-api-access-79r7w\") pod \"maas-api-revoke-sub-405a2ea2-xl97n\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.355944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.355865    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-405a2ea2-xl97n\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.355944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.355912    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-405a2ea2-xl97n\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.454235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.450895    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:57.473865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.471821    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79r7w\" (UniqueName: \"kubernetes.io/projected/75736161-30da-4f0e-9e00-d5c951ea1161-kube-api-access-79r7w\") pod \"maas-api-revoke-sub-405a2ea2-xl97n\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.473865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.472024    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-405a2ea2-xl97n\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.473865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.472064    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-405a2ea2-xl97n\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.473865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.473251    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-405a2ea2-xl97n\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.477315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.477283    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-405a2ea2-xl97n\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.483516 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.483483    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79r7w\" (UniqueName: \"kubernetes.io/projected/75736161-30da-4f0e-9e00-d5c951ea1161-kube-api-access-79r7w\") pod \"maas-api-revoke-sub-405a2ea2-xl97n\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.602373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.602295    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:36:57.914125 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:36:57.914093    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75736161_30da_4f0e_9e00_d5c951ea1161.slice/crio-7a7b257b4077ac3a0bf08217da239f4304f116966cf511af26a9325e9f904b9d WatchSource:0}: Error finding container 7a7b257b4077ac3a0bf08217da239f4304f116966cf511af26a9325e9f904b9d: Status 404 returned error can't find the container with id 7a7b257b4077ac3a0bf08217da239f4304f116966cf511af26a9325e9f904b9d
Nov 03 13:36:57.914777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:57.914656    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"]
Nov 03 13:36:58.007042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:58.007008    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:58.390059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:58.390028    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n" event={"ID":"75736161-30da-4f0e-9e00-d5c951ea1161","Type":"ContainerStarted","Data":"7a7b257b4077ac3a0bf08217da239f4304f116966cf511af26a9325e9f904b9d"}
Nov 03 13:36:58.638957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:58.638922    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:59.357628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:59.357582    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:59.407970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:59.405967    5420 generic.go:356] "Generic (PLEG): container finished" podID="75736161-30da-4f0e-9e00-d5c951ea1161" containerID="72554307c7c5274691c8498ec6aa33dc60f004cd9553c55fd86dbf09bf32c929" exitCode=0
Nov 03 13:36:59.407970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:36:59.406163    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n" event={"ID":"75736161-30da-4f0e-9e00-d5c951ea1161","Type":"ContainerDied","Data":"72554307c7c5274691c8498ec6aa33dc60f004cd9553c55fd86dbf09bf32c929"}
Nov 03 13:37:00.033680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.033651    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:00.606461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.606380    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:00.611615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.611586    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:37:00.729968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.729941    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-openshift-ca\") pod \"75736161-30da-4f0e-9e00-d5c951ea1161\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") "
Nov 03 13:37:00.730149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.729987    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75736161-30da-4f0e-9e00-d5c951ea1161-kube-api-access-79r7w\" (UniqueName: \"kubernetes.io/projected/75736161-30da-4f0e-9e00-d5c951ea1161-kube-api-access-79r7w\") pod \"75736161-30da-4f0e-9e00-d5c951ea1161\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") "
Nov 03 13:37:00.730149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.730119    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-xks-ca\") pod \"75736161-30da-4f0e-9e00-d5c951ea1161\" (UID: \"75736161-30da-4f0e-9e00-d5c951ea1161\") "
Nov 03 13:37:00.730437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.730408    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-openshift-ca" pod "75736161-30da-4f0e-9e00-d5c951ea1161" (UID: "75736161-30da-4f0e-9e00-d5c951ea1161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:37:00.731051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.731023    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:00.732367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.732343    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-xks-ca" pod "75736161-30da-4f0e-9e00-d5c951ea1161" (UID: "75736161-30da-4f0e-9e00-d5c951ea1161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:37:00.732457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.732386    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75736161-30da-4f0e-9e00-d5c951ea1161-kube-api-access-79r7w" pod "75736161-30da-4f0e-9e00-d5c951ea1161" (UID: "75736161-30da-4f0e-9e00-d5c951ea1161"). InnerVolumeSpecName "kube-api-access-79r7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:37:00.832843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.832809    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75736161-30da-4f0e-9e00-d5c951ea1161-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:00.832843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:00.832839    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79r7w\" (UniqueName: \"kubernetes.io/projected/75736161-30da-4f0e-9e00-d5c951ea1161-kube-api-access-79r7w\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:01.414499 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:01.414412    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:01.428952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:01.428911    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n" event={"ID":"75736161-30da-4f0e-9e00-d5c951ea1161","Type":"ContainerDied","Data":"7a7b257b4077ac3a0bf08217da239f4304f116966cf511af26a9325e9f904b9d"}
Nov 03 13:37:01.429119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:01.428976    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a7b257b4077ac3a0bf08217da239f4304f116966cf511af26a9325e9f904b9d"
Nov 03 13:37:01.429119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:01.428932    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"
Nov 03 13:37:02.025531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:02.023270    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:02.624670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:02.624616    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:03.326447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.326412    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:03.849819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.849789    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"]
Nov 03 13:37:03.850527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.850507    5420 cpu_manager.go:414] "containerMap: removing container" podUID="75736161-30da-4f0e-9e00-d5c951ea1161" containerName="revoke-keys"
Nov 03 13:37:03.850527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.850526    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75736161-30da-4f0e-9e00-d5c951ea1161" containerName="revoke-keys"
Nov 03 13:37:03.858251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.852612    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="75736161-30da-4f0e-9e00-d5c951ea1161" containerName="revoke-keys"
Nov 03 13:37:03.860729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.860325    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:03.868522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.868499    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-2b8a08-openshift-default-dockercfg-dffsv\""
Nov 03 13:37:03.888736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.877332    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"]
Nov 03 13:37:03.990087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.990055    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:03.990322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.990093    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:03.990322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.990115    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:03.990322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.990272    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:03.990462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.990417    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:03.990462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.990440    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djk7r\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-kube-api-access-djk7r\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:03.990533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.990470    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:03.990593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.990579    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:03.990630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:03.990610    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.091827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.091775    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.101487    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.101543    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.101631    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.101834    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.101868    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djk7r\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-kube-api-access-djk7r\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.101915    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.102021    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.102062    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.102152    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.102957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.102935    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.104359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.103484    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.104359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.105070    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.114814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.112091    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.114814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.112168    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.114814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.112379    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.114814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.114808    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:04.125684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.125644    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djk7r\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-kube-api-access-djk7r\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.135244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.134294    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.199000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.198967    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:04.458257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.458195    5420 generic.go:356] "Generic (PLEG): container finished" podID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerID="c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44" exitCode=0
Nov 03 13:37:04.458257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.458249    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" event={"ID":"4c550cb4-9c69-4cf7-a9c8-560011a88b56","Type":"ContainerDied","Data":"c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44"}
Nov 03 13:37:04.458790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.458772    5420 scope.go:122] "RemoveContainer" containerID="c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44"
Nov 03 13:37:04.460547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.460519    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:37:04.460658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.460587    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:37:04.460658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.460619    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:37:04.492166 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:37:04.492137    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02f66f5_b9f5_4cf4_bb36_dbf981efb16f.slice/crio-15375beb2f4cd19da26990a0c34bf2e35f6ba1862bc1f8ed2cb1323fd26bfd36 WatchSource:0}: Error finding container 15375beb2f4cd19da26990a0c34bf2e35f6ba1862bc1f8ed2cb1323fd26bfd36: Status 404 returned error can't find the container with id 15375beb2f4cd19da26990a0c34bf2e35f6ba1862bc1f8ed2cb1323fd26bfd36
Nov 03 13:37:04.496080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.496041    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:37:04.496285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.496126    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:37:04.496285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.496170    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:37:04.505007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.504979    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"]
Nov 03 13:37:04.921448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:04.920735    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:05.480095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:05.480058    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" event={"ID":"4c550cb4-9c69-4cf7-a9c8-560011a88b56","Type":"ContainerStarted","Data":"99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9"}
Nov 03 13:37:05.481939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:05.481916    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" event={"ID":"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f","Type":"ContainerStarted","Data":"3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1"}
Nov 03 13:37:05.482038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:05.481960    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" event={"ID":"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f","Type":"ContainerStarted","Data":"15375beb2f4cd19da26990a0c34bf2e35f6ba1862bc1f8ed2cb1323fd26bfd36"}
Nov 03 13:37:05.548236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:05.548157    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:05.559238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:05.552987    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:37:05.559238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:05.553112    5420 patch_prober.go:29] interesting pod/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 03 13:37:05.559238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:05.553163    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 03 13:37:05.599054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:05.598999    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" podStartSLOduration=2.598984467 podStartE2EDuration="2.598984467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:05.579734051 +0000 UTC m=+1666.657499078" watchObservedRunningTime="2026-11-03 13:37:05.598984467 +0000 UTC m=+1666.676749492"
Nov 03 13:37:06.200289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:06.200256    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:06.201359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:06.201341    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 13:37:06.201415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:06.201381    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 13:37:06.284235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:06.283238    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:06.582342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:06.581108    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:06.651243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:06.642312    5420 patch_prober.go:29] interesting pod/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 03 13:37:06.651243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:06.642394    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 03 13:37:07.076079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:07.075292    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"]
Nov 03 13:37:07.116065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:07.116022    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd6923-znkk8"]
Nov 03 13:37:07.205711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:07.205682    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 13:37:07.206024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:07.205996    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 13:37:07.571241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:07.564802    5420 patch_prober.go:29] interesting pod/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 03 13:37:07.571241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:07.564854    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 03 13:37:08.075819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:08.075369    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:08.199506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:08.199470    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 13:37:08.199659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:08.199521    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 13:37:08.205316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:08.205292    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:08.447165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:08.441820    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:08.555913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:08.555876    5420 patch_prober.go:29] interesting pod/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 03 13:37:08.574872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:08.574833    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 03 13:37:09.089043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:09.089010    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465ea315-d67b-499d-828b-d020e913be4b" path="/var/lib/kubelet/pods/465ea315-d67b-499d-828b-d020e913be4b/volumes"
Nov 03 13:37:09.104039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:09.103998    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:09.200145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:09.200103    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 13:37:09.200445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:09.200415    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 13:37:09.553235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:09.552598    5420 patch_prober.go:29] interesting pod/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 03 13:37:09.553235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:09.552658    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 03 13:37:09.735761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:09.735048    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:10.201722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:10.201667    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 13:37:10.202125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:10.201745    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 13:37:10.512736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:10.512698    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:10.556696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:10.556654    5420 patch_prober.go:29] interesting pod/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 03 13:37:10.557010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:10.556959    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 03 13:37:11.203927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:11.203898    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:11.204438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:11.204423    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:11.205241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:11.205186    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:37:11.252537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:11.252505    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:11.556511 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:11.556435    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:37:11.556948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:11.556928    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:37:11.557726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:11.557708    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:37:11.952426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:11.943990    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:12.094389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:12.094345    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:12.611458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:12.609835    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:13.315355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:13.315323    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:13.852920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:13.852891    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:13.993742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:13.993712    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:14.527580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:14.523849    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:15.198648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:15.198602    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:15.957579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:15.957522    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:16.533917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:16.533880    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:17.145520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:17.145488    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:17.610995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:17.610959    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:37:17.817343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:17.817242    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:18.048153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:18.048118    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"]
Nov 03 13:37:18.055425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:18.055400    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82954635-4c89b"]
Nov 03 13:37:18.716159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:18.716109    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:19.093286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:19.093172    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f129cc1f-30b8-4a3b-af67-1db7069e66d1" path="/var/lib/kubelet/pods/f129cc1f-30b8-4a3b-af67-1db7069e66d1/volumes"
Nov 03 13:37:19.159056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:19.159024    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:19.323924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:19.323881    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:19.482990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:19.482931    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:20.698344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:20.695097    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:20.897238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:20.894312    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:37:20.897238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:20.894376    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:37:21.577412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:21.577376    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:37:21.577736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:21.577699    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:37:21.693497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:21.692087    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:21.968924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:21.968890    5420 scope.go:122] "RemoveContainer" containerID="0d35bdb40978c459b29fece20557afb1bafc1969f08375cb38500b61278cecff"
Nov 03 13:37:22.378958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:22.378856    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:23.013505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:23.013438    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:23.718433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:23.718347    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:24.455550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:24.455456    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:24.733348 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:37:24.733265    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:37:25.046148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:25.046060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:25.048387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:25.048357    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"]
Nov 03 13:37:25.052895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:25.052873    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe9b86c9-7vh99"]
Nov 03 13:37:25.088366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:25.088333    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157c9882-fa19-4a48-8e54-ad5ea40d0364" path="/var/lib/kubelet/pods/157c9882-fa19-4a48-8e54-ad5ea40d0364/volumes"
Nov 03 13:37:25.795256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:25.794265    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:26.531957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:26.531456    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:27.210512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:27.210479    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:27.819634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:27.819340    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:28.441226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:28.441177    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:29.130103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:29.130063    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:29.742911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:29.742875    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:30.418090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:30.418054    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:31.208630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:31.208596    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:31.950726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:31.950693    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:32.509937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:32.509900    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:33.158612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:33.158566    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:34.035960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:34.035923    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:34.724200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:34.724164    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:35.346102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:35.346060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:35.948638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:35.948551    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:36.516627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:36.516579    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:37.142407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:37.142070    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:37.722005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:37.719509    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:38.330848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:38.330816    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:39.032236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:39.030808    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:39.851845 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:39.851813    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:40.335647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:40.312751    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:41.027663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:41.027625    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:41.618998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:41.618954    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:42.298666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:42.298625    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:42.954544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:42.954508    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:43.545234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:43.544206    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:44.310264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:44.310178    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:44.918408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:44.916180    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:45.532769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:45.532726    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:46.127344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:46.127289    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:46.716642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:46.716170    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:47.315486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:47.315449    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:47.904706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:47.904669    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:48.531840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:48.530993    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:48.672755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:48.672724    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"]
Nov 03 13:37:48.688751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:48.688721    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"]
Nov 03 13:37:48.797630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:48.797547    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:48.948124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:48.948088    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:49.528333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:49.528300    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:50.198729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:50.198685    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:50.851317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:50.851287    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:51.108721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:51.108621    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:51.695140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:51.695102    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:52.326630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:52.326516    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:52.916715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:52.916681    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:53.561594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:53.561560    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:53.673380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:53.673324    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g" podUID="482e5ea4-ff10-4559-8299-4e0bf09c39d0" containerName="payload-processing" containerID="cri-o://5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10" gracePeriod=25
Nov 03 13:37:53.690062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:53.690029    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr" podUID="f43ea246-19cd-4466-bc4e-f7eecb3b22fc" containerName="payload-pre-processing" containerID="cri-o://5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7" gracePeriod=25
Nov 03 13:37:53.960733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:53.960651    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:54.046686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.046662    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:37:54.070546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.070450    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:37:54.129297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.125798    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482e5ea4-ff10-4559-8299-4e0bf09c39d0-kube-api-access-gjqhl\" (UniqueName: \"kubernetes.io/projected/482e5ea4-ff10-4559-8299-4e0bf09c39d0-kube-api-access-gjqhl\") pod \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\" (UID: \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\") "
Nov 03 13:37:54.129297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.125895    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482e5ea4-ff10-4559-8299-4e0bf09c39d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482e5ea4-ff10-4559-8299-4e0bf09c39d0-plugins-config-volume\") pod \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\" (UID: \"482e5ea4-ff10-4559-8299-4e0bf09c39d0\") "
Nov 03 13:37:54.129570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.129457    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482e5ea4-ff10-4559-8299-4e0bf09c39d0-plugins-config-volume" pod "482e5ea4-ff10-4559-8299-4e0bf09c39d0" (UID: "482e5ea4-ff10-4559-8299-4e0bf09c39d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:37:54.150189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.149875    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482e5ea4-ff10-4559-8299-4e0bf09c39d0-kube-api-access-gjqhl" pod "482e5ea4-ff10-4559-8299-4e0bf09c39d0" (UID: "482e5ea4-ff10-4559-8299-4e0bf09c39d0"). InnerVolumeSpecName "kube-api-access-gjqhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:37:54.253527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.238024    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-plugins-config-volume\") pod \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\" (UID: \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\") "
Nov 03 13:37:54.253527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.238255    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-kube-api-access-pncnx\" (UniqueName: \"kubernetes.io/projected/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-kube-api-access-pncnx\") pod \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\" (UID: \"f43ea246-19cd-4466-bc4e-f7eecb3b22fc\") "
Nov 03 13:37:54.253527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.239020    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjqhl\" (UniqueName: \"kubernetes.io/projected/482e5ea4-ff10-4559-8299-4e0bf09c39d0-kube-api-access-gjqhl\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:54.253527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.239048    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482e5ea4-ff10-4559-8299-4e0bf09c39d0-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:54.253527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.251495    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-plugins-config-volume" pod "f43ea246-19cd-4466-bc4e-f7eecb3b22fc" (UID: "f43ea246-19cd-4466-bc4e-f7eecb3b22fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:37:54.253527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.251634    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-kube-api-access-pncnx" pod "f43ea246-19cd-4466-bc4e-f7eecb3b22fc" (UID: "f43ea246-19cd-4466-bc4e-f7eecb3b22fc"). InnerVolumeSpecName "kube-api-access-pncnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:37:54.279969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.270505    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:54.284262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.280767    5420 generic.go:356] "Generic (PLEG): container finished" podID="f43ea246-19cd-4466-bc4e-f7eecb3b22fc" containerID="5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7" exitCode=0
Nov 03 13:37:54.284262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.280969    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"
Nov 03 13:37:54.293856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.293238    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr" event={"ID":"f43ea246-19cd-4466-bc4e-f7eecb3b22fc","Type":"ContainerDied","Data":"5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7"}
Nov 03 13:37:54.293856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.293639    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr" event={"ID":"f43ea246-19cd-4466-bc4e-f7eecb3b22fc","Type":"ContainerDied","Data":"1d276c453e6bd47d3be1cabf8fdb34bb7997ba613dec3b6548b82c6b78bc96e0"}
Nov 03 13:37:54.293856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.293691    5420 scope.go:122] "RemoveContainer" containerID="5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7"
Nov 03 13:37:54.303138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.297986    5420 generic.go:356] "Generic (PLEG): container finished" podID="482e5ea4-ff10-4559-8299-4e0bf09c39d0" containerID="5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10" exitCode=0
Nov 03 13:37:54.303138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.298242    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g" event={"ID":"482e5ea4-ff10-4559-8299-4e0bf09c39d0","Type":"ContainerDied","Data":"5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10"}
Nov 03 13:37:54.303138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.298282    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g" event={"ID":"482e5ea4-ff10-4559-8299-4e0bf09c39d0","Type":"ContainerDied","Data":"a5d64dc2ff803afa8db1d2a17db106365b8a48d26a5e9e616e1aa40d667ef6ce"}
Nov 03 13:37:54.303138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.298341    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"
Nov 03 13:37:54.311508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.311490    5420 scope.go:122] "RemoveContainer" containerID="5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7"
Nov 03 13:37:54.312030 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:37:54.311979    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7\": container with ID starting with 5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7 not found: ID does not exist" containerID="5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7"
Nov 03 13:37:54.312401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.312043    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7"} err="failed to get container status \"5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7\": rpc error: code = NotFound desc = could not find container \"5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7\": container with ID starting with 5a7984edde8b9984cdc7cf44b83ff27c004bd2527fa507f9ff18e2773e6783b7 not found: ID does not exist"
Nov 03 13:37:54.312401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.312068    5420 scope.go:122] "RemoveContainer" containerID="5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10"
Nov 03 13:37:54.340542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.340512    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pncnx\" (UniqueName: \"kubernetes.io/projected/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-kube-api-access-pncnx\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:54.340542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.340540    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43ea246-19cd-4466-bc4e-f7eecb3b22fc-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:54.340821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.340568    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"]
Nov 03 13:37:54.362074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.346905    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-146318-bcd6dd8cf-mntvr"]
Nov 03 13:37:54.362434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.362277    5420 scope.go:122] "RemoveContainer" containerID="5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10"
Nov 03 13:37:54.364586 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:37:54.362968    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10\": container with ID starting with 5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10 not found: ID does not exist" containerID="5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10"
Nov 03 13:37:54.364586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.363003    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10"} err="failed to get container status \"5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10\": rpc error: code = NotFound desc = could not find container \"5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10\": container with ID starting with 5151ec93eabd05497c8c3649ebf0bc040c0e73541e7a4ab8b1e0b3de12ca9f10 not found: ID does not exist"
Nov 03 13:37:54.366200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.366175    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"]
Nov 03 13:37:54.374862 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:54.374775    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-146318-7c785754ff-s472g"]
Nov 03 13:37:54.732755 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:37:54.732721    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:37:55.002686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:55.002617    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:55.107787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:55.107751    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482e5ea4-ff10-4559-8299-4e0bf09c39d0" path="/var/lib/kubelet/pods/482e5ea4-ff10-4559-8299-4e0bf09c39d0/volumes"
Nov 03 13:37:55.108490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:55.108470    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43ea246-19cd-4466-bc4e-f7eecb3b22fc" path="/var/lib/kubelet/pods/f43ea246-19cd-4466-bc4e-f7eecb3b22fc/volumes"
Nov 03 13:37:55.638811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:55.636717    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:56.183578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.183549    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:56.333990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.333911    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:56.376611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.376575    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"]
Nov 03 13:37:56.376819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.376783    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy" containerID="cri-o://50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135" gracePeriod=30
Nov 03 13:37:56.470107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.470069    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"]
Nov 03 13:37:56.471130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.471113    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f43ea246-19cd-4466-bc4e-f7eecb3b22fc" containerName="payload-pre-processing"
Nov 03 13:37:56.471190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.471133    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43ea246-19cd-4466-bc4e-f7eecb3b22fc" containerName="payload-pre-processing"
Nov 03 13:37:56.471190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.471181    5420 cpu_manager.go:414] "containerMap: removing container" podUID="482e5ea4-ff10-4559-8299-4e0bf09c39d0" containerName="payload-processing"
Nov 03 13:37:56.471272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.471189    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482e5ea4-ff10-4559-8299-4e0bf09c39d0" containerName="payload-processing"
Nov 03 13:37:56.471416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.471404    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43ea246-19cd-4466-bc4e-f7eecb3b22fc" containerName="payload-pre-processing"
Nov 03 13:37:56.471453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.471428    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="482e5ea4-ff10-4559-8299-4e0bf09c39d0" containerName="payload-processing"
Nov 03 13:37:56.478058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.478033    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.478532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.478503    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"]
Nov 03 13:37:56.495623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.495591    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:37:56.495827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.495789    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:37:56.496331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.496312    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:37:56.496475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.496374    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:37:56.598952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.598918    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b7b145c-9ngj8\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.599184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.599156    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b7b145c-9ngj8\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.599406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.599386    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l77zt\" (UniqueName: \"kubernetes.io/projected/0298474e-45a1-4b2b-86b1-cb1fbebb591e-kube-api-access-l77zt\") pod \"maas-api-revoke-sub-1b7b145c-9ngj8\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.701238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.701193    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b7b145c-9ngj8\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.701439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.701358    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b7b145c-9ngj8\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.701523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.701483    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l77zt\" (UniqueName: \"kubernetes.io/projected/0298474e-45a1-4b2b-86b1-cb1fbebb591e-kube-api-access-l77zt\") pod \"maas-api-revoke-sub-1b7b145c-9ngj8\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.702032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.702004    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b7b145c-9ngj8\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.703617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.703595    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b7b145c-9ngj8\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.710041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.710016    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l77zt\" (UniqueName: \"kubernetes.io/projected/0298474e-45a1-4b2b-86b1-cb1fbebb591e-kube-api-access-l77zt\") pod \"maas-api-revoke-sub-1b7b145c-9ngj8\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.816164 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.816074    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:56.902993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:56.900336    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:57.045197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:57.045159    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:57.113826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:57.113795    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"]
Nov 03 13:37:57.113972 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:37:57.113845    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0298474e_45a1_4b2b_86b1_cb1fbebb591e.slice/crio-afe151306b4edb57607fb044610c6494036e8a0f295e1e0ad58d5837f065db06 WatchSource:0}: Error finding container afe151306b4edb57607fb044610c6494036e8a0f295e1e0ad58d5837f065db06: Status 404 returned error can't find the container with id afe151306b4edb57607fb044610c6494036e8a0f295e1e0ad58d5837f065db06
Nov 03 13:37:57.332781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:57.332753    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8" event={"ID":"0298474e-45a1-4b2b-86b1-cb1fbebb591e","Type":"ContainerStarted","Data":"afe151306b4edb57607fb044610c6494036e8a0f295e1e0ad58d5837f065db06"}
Nov 03 13:37:57.405463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:57.405038    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:57.665911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:57.665832    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"]
Nov 03 13:37:57.666131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:57.666103    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy" containerID="cri-o://99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9" gracePeriod=30
Nov 03 13:37:57.792655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:57.792546    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:58.150754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:58.150725    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:58.344052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:58.344004    5420 generic.go:356] "Generic (PLEG): container finished" podID="0298474e-45a1-4b2b-86b1-cb1fbebb591e" containerID="26701db19b4e5241fe6aae47ba48e77b400bd6c9e7d2d0ec19aa8e4d146ac0f4" exitCode=0
Nov 03 13:37:58.344477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:58.344075    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8" event={"ID":"0298474e-45a1-4b2b-86b1-cb1fbebb591e","Type":"ContainerDied","Data":"26701db19b4e5241fe6aae47ba48e77b400bd6c9e7d2d0ec19aa8e4d146ac0f4"}
Nov 03 13:37:58.391375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:58.391301    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:58.759558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:58.759521    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:58.957690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:58.957654    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:59.155994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.155896    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:59.535007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.534985    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:37:59.580278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.578084    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-xks-ca\") pod \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") "
Nov 03 13:37:59.580278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.578293    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-openshift-ca\") pod \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") "
Nov 03 13:37:59.581582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.580708    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-openshift-ca" pod "0298474e-45a1-4b2b-86b1-cb1fbebb591e" (UID: "0298474e-45a1-4b2b-86b1-cb1fbebb591e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:37:59.588821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.587694    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-xks-ca" pod "0298474e-45a1-4b2b-86b1-cb1fbebb591e" (UID: "0298474e-45a1-4b2b-86b1-cb1fbebb591e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:37:59.589615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.589589    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:59.682052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.681970    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0298474e-45a1-4b2b-86b1-cb1fbebb591e-kube-api-access-l77zt\" (UniqueName: \"kubernetes.io/projected/0298474e-45a1-4b2b-86b1-cb1fbebb591e-kube-api-access-l77zt\") pod \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\" (UID: \"0298474e-45a1-4b2b-86b1-cb1fbebb591e\") "
Nov 03 13:37:59.682555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.682536    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:59.682659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.682557    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0298474e-45a1-4b2b-86b1-cb1fbebb591e-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:37:59.684144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.684110    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0298474e-45a1-4b2b-86b1-cb1fbebb591e-kube-api-access-l77zt" pod "0298474e-45a1-4b2b-86b1-cb1fbebb591e" (UID: "0298474e-45a1-4b2b-86b1-cb1fbebb591e"). InnerVolumeSpecName "kube-api-access-l77zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:37:59.784599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:37:59.784562    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l77zt\" (UniqueName: \"kubernetes.io/projected/0298474e-45a1-4b2b-86b1-cb1fbebb591e-kube-api-access-l77zt\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:00.207159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:00.206385    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:00.362059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:00.362014    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8" event={"ID":"0298474e-45a1-4b2b-86b1-cb1fbebb591e","Type":"ContainerDied","Data":"afe151306b4edb57607fb044610c6494036e8a0f295e1e0ad58d5837f065db06"}
Nov 03 13:38:00.362059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:00.362053    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"
Nov 03 13:38:00.362400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:00.362077    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afe151306b4edb57607fb044610c6494036e8a0f295e1e0ad58d5837f065db06"
Nov 03 13:38:00.883272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:00.878856    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:01.261839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.261804    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"]
Nov 03 13:38:01.286321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.284785    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"]
Nov 03 13:38:01.397340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.397292    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:01.543998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.543898    5420 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 13:38:01.543998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.543981    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 13:38:01.544528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.544503    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"]
Nov 03 13:38:01.544824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.544792    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy" containerID="cri-o://2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611" gracePeriod=30
Nov 03 13:38:01.609400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.600441    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:01.895293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.894370    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:01.973665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.973635    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:38:01.988631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.988596    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istiod-ca-cert\") pod \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") "
Nov 03 13:38:01.988826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.988653    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-podinfo\") pod \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") "
Nov 03 13:38:01.988826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.988691    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-data\") pod \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") "
Nov 03 13:38:01.988826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.988720    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-envoy\") pod \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") "
Nov 03 13:38:01.988826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.988791    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-credential-socket\") pod \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") "
Nov 03 13:38:01.989070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.988880    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-kube-api-access-db9hx\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-kube-api-access-db9hx\") pod \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") "
Nov 03 13:38:01.989070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.988919    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-certs\") pod \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") "
Nov 03 13:38:01.989070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.988943    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-socket\") pod \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") "
Nov 03 13:38:01.989070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.988961    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-token\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-token\") pod \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\" (UID: \"15be20d9-c7e8-4696-a9cb-3395faffbd9d\") "
Nov 03 13:38:01.989070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.989014    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istiod-ca-cert" pod "15be20d9-c7e8-4696-a9cb-3395faffbd9d" (UID: "15be20d9-c7e8-4696-a9cb-3395faffbd9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:01.989607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.989362    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-data" pod "15be20d9-c7e8-4696-a9cb-3395faffbd9d" (UID: "15be20d9-c7e8-4696-a9cb-3395faffbd9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:01.989607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.989509    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-certs" pod "15be20d9-c7e8-4696-a9cb-3395faffbd9d" (UID: "15be20d9-c7e8-4696-a9cb-3395faffbd9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:01.989607 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.989562    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-socket" pod "15be20d9-c7e8-4696-a9cb-3395faffbd9d" (UID: "15be20d9-c7e8-4696-a9cb-3395faffbd9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:01.989785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.989764    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-credential-socket" pod "15be20d9-c7e8-4696-a9cb-3395faffbd9d" (UID: "15be20d9-c7e8-4696-a9cb-3395faffbd9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:01.989962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.989943    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:01.990037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.989965    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:01.990037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.989980    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:01.990037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.989993    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:01.990037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.990007    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:01.991980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.991924    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-kube-api-access-db9hx" pod "15be20d9-c7e8-4696-a9cb-3395faffbd9d" (UID: "15be20d9-c7e8-4696-a9cb-3395faffbd9d"). InnerVolumeSpecName "kube-api-access-db9hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:01.992176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.992038    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-podinfo" pod "15be20d9-c7e8-4696-a9cb-3395faffbd9d" (UID: "15be20d9-c7e8-4696-a9cb-3395faffbd9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:38:01.993664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.993638    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-token" pod "15be20d9-c7e8-4696-a9cb-3395faffbd9d" (UID: "15be20d9-c7e8-4696-a9cb-3395faffbd9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:01.993784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:01.993696    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-envoy" pod "15be20d9-c7e8-4696-a9cb-3395faffbd9d" (UID: "15be20d9-c7e8-4696-a9cb-3395faffbd9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:02.092205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.092165    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:02.092205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.092205    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:02.092430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.092238    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db9hx\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-kube-api-access-db9hx\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:02.092430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.092253    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15be20d9-c7e8-4696-a9cb-3395faffbd9d-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:02.373512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.373474    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:38:02.383335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.383303    5420 generic.go:356] "Generic (PLEG): container finished" podID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerID="50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135" exitCode=0
Nov 03 13:38:02.383540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.383372    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"
Nov 03 13:38:02.383540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.383373    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" event={"ID":"15be20d9-c7e8-4696-a9cb-3395faffbd9d","Type":"ContainerDied","Data":"50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135"}
Nov 03 13:38:02.383540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.383512    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6" event={"ID":"15be20d9-c7e8-4696-a9cb-3395faffbd9d","Type":"ContainerDied","Data":"e113b91f41902a267f4b676cc41e1e23a8b3eb30245ad78dfebc241ada948943"}
Nov 03 13:38:02.383719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.383550    5420 scope.go:122] "RemoveContainer" containerID="50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135"
Nov 03 13:38:02.397483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.397461    5420 scope.go:122] "RemoveContainer" containerID="50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135"
Nov 03 13:38:02.397824 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:38:02.397800    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135\": container with ID starting with 50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135 not found: ID does not exist" containerID="50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135"
Nov 03 13:38:02.397925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.397833    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135"} err="failed to get container status \"50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135\": rpc error: code = NotFound desc = could not find container \"50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135\": container with ID starting with 50d055543250345a8f71277d338c357c92932800d3c028f9b49c124fb99b8135 not found: ID does not exist"
Nov 03 13:38:02.421184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.421145    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"]
Nov 03 13:38:02.425896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.425871    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-799v6"]
Nov 03 13:38:02.933875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:02.932416    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:03.032460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.032436    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:38:03.088247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.088196    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" path="/var/lib/kubelet/pods/15be20d9-c7e8-4696-a9cb-3395faffbd9d/volumes"
Nov 03 13:38:03.104752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.104705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:03.128604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.128571    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-kube-api-access-xx2fs\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-kube-api-access-xx2fs\") pod \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") "
Nov 03 13:38:03.128799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.128617    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-credential-socket\") pod \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") "
Nov 03 13:38:03.128799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.128694    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-data\") pod \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") "
Nov 03 13:38:03.128799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.128746    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-token\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-token\") pod \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") "
Nov 03 13:38:03.128799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.128781    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-certs\") pod \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") "
Nov 03 13:38:03.129038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.128832    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-socket\") pod \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") "
Nov 03 13:38:03.129038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.128887    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istiod-ca-cert\") pod \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") "
Nov 03 13:38:03.129038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.128985    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-credential-socket" pod "4c550cb4-9c69-4cf7-a9c8-560011a88b56" (UID: "4c550cb4-9c69-4cf7-a9c8-560011a88b56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:03.129193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.129012    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-data" pod "4c550cb4-9c69-4cf7-a9c8-560011a88b56" (UID: "4c550cb4-9c69-4cf7-a9c8-560011a88b56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:03.129193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.129025    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-podinfo\") pod \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") "
Nov 03 13:38:03.129193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.129098    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-certs" pod "4c550cb4-9c69-4cf7-a9c8-560011a88b56" (UID: "4c550cb4-9c69-4cf7-a9c8-560011a88b56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:03.129193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.129141    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-envoy\") pod \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\" (UID: \"4c550cb4-9c69-4cf7-a9c8-560011a88b56\") "
Nov 03 13:38:03.129450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.129263    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-socket" pod "4c550cb4-9c69-4cf7-a9c8-560011a88b56" (UID: "4c550cb4-9c69-4cf7-a9c8-560011a88b56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:03.129450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.129358    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istiod-ca-cert" pod "4c550cb4-9c69-4cf7-a9c8-560011a88b56" (UID: "4c550cb4-9c69-4cf7-a9c8-560011a88b56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:03.130011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.129988    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:03.130070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.130019    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:03.130070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.130034    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:03.130070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.130047    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:03.130070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.130061    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:03.131067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.131039    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-kube-api-access-xx2fs" pod "4c550cb4-9c69-4cf7-a9c8-560011a88b56" (UID: "4c550cb4-9c69-4cf7-a9c8-560011a88b56"). InnerVolumeSpecName "kube-api-access-xx2fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:03.131170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.131065    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-podinfo" pod "4c550cb4-9c69-4cf7-a9c8-560011a88b56" (UID: "4c550cb4-9c69-4cf7-a9c8-560011a88b56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:38:03.131514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.131494    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-envoy" pod "4c550cb4-9c69-4cf7-a9c8-560011a88b56" (UID: "4c550cb4-9c69-4cf7-a9c8-560011a88b56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:03.131624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.131609    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-token" pod "4c550cb4-9c69-4cf7-a9c8-560011a88b56" (UID: "4c550cb4-9c69-4cf7-a9c8-560011a88b56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:03.231686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.231651    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx2fs\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-kube-api-access-xx2fs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:03.231686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.231680    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:03.231686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.231690    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:03.231955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.231699    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c550cb4-9c69-4cf7-a9c8-560011a88b56-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:03.291760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.291718    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:03.393243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.393178    5420 generic.go:356] "Generic (PLEG): container finished" podID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerID="99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9" exitCode=0
Nov 03 13:38:03.393441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.393249    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" event={"ID":"4c550cb4-9c69-4cf7-a9c8-560011a88b56","Type":"ContainerDied","Data":"99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9"}
Nov 03 13:38:03.393441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.393270    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"
Nov 03 13:38:03.393441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.393313    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk" event={"ID":"4c550cb4-9c69-4cf7-a9c8-560011a88b56","Type":"ContainerDied","Data":"2c953ff97b7aac59766e6609c82b05e16232512e7ecaf5953612345b46ff0fae"}
Nov 03 13:38:03.393441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.393341    5420 scope.go:122] "RemoveContainer" containerID="99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9"
Nov 03 13:38:03.405573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.405553    5420 scope.go:122] "RemoveContainer" containerID="c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44"
Nov 03 13:38:03.417873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.417649    5420 scope.go:122] "RemoveContainer" containerID="99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9"
Nov 03 13:38:03.419827 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:38:03.419805    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9\": container with ID starting with 99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9 not found: ID does not exist" containerID="99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9"
Nov 03 13:38:03.419937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.419835    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9"} err="failed to get container status \"99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9\": rpc error: code = NotFound desc = could not find container \"99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9\": container with ID starting with 99c492da1e16e6912e949a7a3f9aa52247a571aef59f30f2b39bcec611b134f9 not found: ID does not exist"
Nov 03 13:38:03.419937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.419853    5420 scope.go:122] "RemoveContainer" containerID="c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44"
Nov 03 13:38:03.420185 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:38:03.420166    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44\": container with ID starting with c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44 not found: ID does not exist" containerID="c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44"
Nov 03 13:38:03.420309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.420192    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44"} err="failed to get container status \"c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44\": rpc error: code = NotFound desc = could not find container \"c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44\": container with ID starting with c11cb99a5df985f3de42f8d8782903f9cf59c328c056c8f5cf8f40c218803f44 not found: ID does not exist"
Nov 03 13:38:03.436417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.436388    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"]
Nov 03 13:38:03.453051 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:03.453022    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-146318-openshift-default-85cf4bdd97-gfnwk"]
Nov 03 13:38:04.147820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.147776    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:04.579732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.579697    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:04.633611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.633575    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"]
Nov 03 13:38:04.634665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634639    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0298474e-45a1-4b2b-86b1-cb1fbebb591e" containerName="revoke-keys"
Nov 03 13:38:04.634665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634665    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0298474e-45a1-4b2b-86b1-cb1fbebb591e" containerName="revoke-keys"
Nov 03 13:38:04.634832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634693    5420 cpu_manager.go:414] "containerMap: removing container" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy"
Nov 03 13:38:04.634832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634703    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy"
Nov 03 13:38:04.634832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634762    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy"
Nov 03 13:38:04.634832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634772    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy"
Nov 03 13:38:04.634832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634787    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy"
Nov 03 13:38:04.634832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634794    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy"
Nov 03 13:38:04.635042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634968    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy"
Nov 03 13:38:04.635042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.634981    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" containerName="istio-proxy"
Nov 03 13:38:04.635042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.635017    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0298474e-45a1-4b2b-86b1-cb1fbebb591e" containerName="revoke-keys"
Nov 03 13:38:04.635042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.635029    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="15be20d9-c7e8-4696-a9cb-3395faffbd9d" containerName="istio-proxy"
Nov 03 13:38:04.641002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.640973    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"]
Nov 03 13:38:04.641155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.641137    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:04.645653 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.645633    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:04.649903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.649881    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-6f33a0\""
Nov 03 13:38:04.650055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.649934    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-6f33a0-dockercfg-mjmr2\""
Nov 03 13:38:04.653703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.653392    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"]
Nov 03 13:38:04.666172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.666135    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"]
Nov 03 13:38:04.742579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.742388    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:04.818239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.806325    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b55x2\" (UniqueName: \"kubernetes.io/projected/30568271-6049-4af4-bf3a-39659d975931-kube-api-access-b55x2\") pod \"payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4\" (UID: \"30568271-6049-4af4-bf3a-39659d975931\") " pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:04.818239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.811452    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30568271-6049-4af4-bf3a-39659d975931-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4\" (UID: \"30568271-6049-4af4-bf3a-39659d975931\") " pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:04.818239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.811782    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539260d3-0a89-434f-a1a5-44890179b602-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g\" (UID: \"539260d3-0a89-434f-a1a5-44890179b602\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:04.818239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.815347    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckdzb\" (UniqueName: \"kubernetes.io/projected/539260d3-0a89-434f-a1a5-44890179b602-kube-api-access-ckdzb\") pod \"payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g\" (UID: \"539260d3-0a89-434f-a1a5-44890179b602\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:04.930039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.926551    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539260d3-0a89-434f-a1a5-44890179b602-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g\" (UID: \"539260d3-0a89-434f-a1a5-44890179b602\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:04.930039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.926602    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckdzb\" (UniqueName: \"kubernetes.io/projected/539260d3-0a89-434f-a1a5-44890179b602-kube-api-access-ckdzb\") pod \"payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g\" (UID: \"539260d3-0a89-434f-a1a5-44890179b602\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:04.930039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.926738    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b55x2\" (UniqueName: \"kubernetes.io/projected/30568271-6049-4af4-bf3a-39659d975931-kube-api-access-b55x2\") pod \"payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4\" (UID: \"30568271-6049-4af4-bf3a-39659d975931\") " pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:04.930039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.926829    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30568271-6049-4af4-bf3a-39659d975931-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4\" (UID: \"30568271-6049-4af4-bf3a-39659d975931\") " pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:04.930039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.927915    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30568271-6049-4af4-bf3a-39659d975931-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4\" (UID: \"30568271-6049-4af4-bf3a-39659d975931\") " pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:04.930039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.928693    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539260d3-0a89-434f-a1a5-44890179b602-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g\" (UID: \"539260d3-0a89-434f-a1a5-44890179b602\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:04.971279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.967923    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b55x2\" (UniqueName: \"kubernetes.io/projected/30568271-6049-4af4-bf3a-39659d975931-kube-api-access-b55x2\") pod \"payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4\" (UID: \"30568271-6049-4af4-bf3a-39659d975931\") " pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:04.998014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.986601    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:04.998014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:04.988028    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckdzb\" (UniqueName: \"kubernetes.io/projected/539260d3-0a89-434f-a1a5-44890179b602-kube-api-access-ckdzb\") pod \"payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g\" (UID: \"539260d3-0a89-434f-a1a5-44890179b602\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:05.091333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:05.091300    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c550cb4-9c69-4cf7-a9c8-560011a88b56" path="/var/lib/kubelet/pods/4c550cb4-9c69-4cf7-a9c8-560011a88b56/volumes"
Nov 03 13:38:05.215293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:05.203268    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:05.296930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:05.296878    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:05.327105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:05.327069    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"]
Nov 03 13:38:05.331399 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:38:05.331336    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30568271_6049_4af4_bf3a_39659d975931.slice/crio-a784ade5bb34f061fb0781a6b248b5de103a99c1e5042051c581e75be8c96323 WatchSource:0}: Error finding container a784ade5bb34f061fb0781a6b248b5de103a99c1e5042051c581e75be8c96323: Status 404 returned error can't find the container with id a784ade5bb34f061fb0781a6b248b5de103a99c1e5042051c581e75be8c96323
Nov 03 13:38:05.435104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:05.435064    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4" event={"ID":"30568271-6049-4af4-bf3a-39659d975931","Type":"ContainerStarted","Data":"a784ade5bb34f061fb0781a6b248b5de103a99c1e5042051c581e75be8c96323"}
Nov 03 13:38:05.629898 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:38:05.629869    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539260d3_0a89_434f_a1a5_44890179b602.slice/crio-60df2f3c9678ae7ee938b7433181ce760fa69d652c58112a0b9237c5e0a99a5c WatchSource:0}: Error finding container 60df2f3c9678ae7ee938b7433181ce760fa69d652c58112a0b9237c5e0a99a5c: Status 404 returned error can't find the container with id 60df2f3c9678ae7ee938b7433181ce760fa69d652c58112a0b9237c5e0a99a5c
Nov 03 13:38:05.631183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:05.631156    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"]
Nov 03 13:38:05.827325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:05.827292    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:06.262368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.262315    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg" podUID="bae77e87-e96d-45d4-a495-7dc77e619250" containerName="payload-processing" containerID="cri-o://41196dcba1b59bf70ac9ff3736e59f9745e308dc8d043bb00a1c60ad74123d5f" gracePeriod=25
Nov 03 13:38:06.285718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.285688    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h" podUID="4b6c86e0-9ece-4256-8456-8879c6782878" containerName="payload-pre-processing" containerID="cri-o://a28d98170c8a25070fa2f596a848dc13dba94eb563b642d97567239d66c185c1" gracePeriod=25
Nov 03 13:38:06.453129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.452305    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g" event={"ID":"539260d3-0a89-434f-a1a5-44890179b602","Type":"ContainerStarted","Data":"f8250ef7498a596657730522b435a98cd102ca5169f699d83a6a1edb0e51e7e8"}
Nov 03 13:38:06.453129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.452375    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g" event={"ID":"539260d3-0a89-434f-a1a5-44890179b602","Type":"ContainerStarted","Data":"60df2f3c9678ae7ee938b7433181ce760fa69d652c58112a0b9237c5e0a99a5c"}
Nov 03 13:38:06.453129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.452403    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:06.455415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.455383    5420 generic.go:356] "Generic (PLEG): container finished" podID="bae77e87-e96d-45d4-a495-7dc77e619250" containerID="41196dcba1b59bf70ac9ff3736e59f9745e308dc8d043bb00a1c60ad74123d5f" exitCode=0
Nov 03 13:38:06.455536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.455462    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg" event={"ID":"bae77e87-e96d-45d4-a495-7dc77e619250","Type":"ContainerDied","Data":"41196dcba1b59bf70ac9ff3736e59f9745e308dc8d043bb00a1c60ad74123d5f"}
Nov 03 13:38:06.459418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.459139    5420 generic.go:356] "Generic (PLEG): container finished" podID="4b6c86e0-9ece-4256-8456-8879c6782878" containerID="a28d98170c8a25070fa2f596a848dc13dba94eb563b642d97567239d66c185c1" exitCode=0
Nov 03 13:38:06.459418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.459190    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h" event={"ID":"4b6c86e0-9ece-4256-8456-8879c6782878","Type":"ContainerDied","Data":"a28d98170c8a25070fa2f596a848dc13dba94eb563b642d97567239d66c185c1"}
Nov 03 13:38:06.462085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.461785    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4" event={"ID":"30568271-6049-4af4-bf3a-39659d975931","Type":"ContainerStarted","Data":"0fe3953c5f6f965362c9d9df365f7c3c673f9a4b20e04f10c874fb609f423a02"}
Nov 03 13:38:06.462085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.461917    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:06.494990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.494920    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g" podStartSLOduration=2.494903397 podStartE2EDuration="2.494903397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:06.48903992 +0000 UTC m=+1727.566804949" watchObservedRunningTime="2026-11-03 13:38:06.494903397 +0000 UTC m=+1727.572668424"
Nov 03 13:38:06.534732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.534645    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4" podStartSLOduration=2.53463067 podStartE2EDuration="2.53463067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:06.532138189 +0000 UTC m=+1727.609903217" watchObservedRunningTime="2026-11-03 13:38:06.53463067 +0000 UTC m=+1727.612395728"
Nov 03 13:38:06.634970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.634935    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:06.690069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.689470    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:38:06.694500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.694481    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:38:06.777749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.777719    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:06.783779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.783752    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae77e87-e96d-45d4-a495-7dc77e619250-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae77e87-e96d-45d4-a495-7dc77e619250-plugins-config-volume\") pod \"bae77e87-e96d-45d4-a495-7dc77e619250\" (UID: \"bae77e87-e96d-45d4-a495-7dc77e619250\") "
Nov 03 13:38:06.783914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.783829    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae77e87-e96d-45d4-a495-7dc77e619250-kube-api-access-dkdfw\" (UniqueName: \"kubernetes.io/projected/bae77e87-e96d-45d4-a495-7dc77e619250-kube-api-access-dkdfw\") pod \"bae77e87-e96d-45d4-a495-7dc77e619250\" (UID: \"bae77e87-e96d-45d4-a495-7dc77e619250\") "
Nov 03 13:38:06.784032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.784004    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6c86e0-9ece-4256-8456-8879c6782878-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6c86e0-9ece-4256-8456-8879c6782878-plugins-config-volume\") pod \"4b6c86e0-9ece-4256-8456-8879c6782878\" (UID: \"4b6c86e0-9ece-4256-8456-8879c6782878\") "
Nov 03 13:38:06.784243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.784186    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6c86e0-9ece-4256-8456-8879c6782878-kube-api-access-hcfxp\" (UniqueName: \"kubernetes.io/projected/4b6c86e0-9ece-4256-8456-8879c6782878-kube-api-access-hcfxp\") pod \"4b6c86e0-9ece-4256-8456-8879c6782878\" (UID: \"4b6c86e0-9ece-4256-8456-8879c6782878\") "
Nov 03 13:38:06.784315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.784238    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae77e87-e96d-45d4-a495-7dc77e619250-plugins-config-volume" pod "bae77e87-e96d-45d4-a495-7dc77e619250" (UID: "bae77e87-e96d-45d4-a495-7dc77e619250"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:06.784713 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.784687    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6c86e0-9ece-4256-8456-8879c6782878-plugins-config-volume" pod "4b6c86e0-9ece-4256-8456-8879c6782878" (UID: "4b6c86e0-9ece-4256-8456-8879c6782878"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:06.785053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.784966    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae77e87-e96d-45d4-a495-7dc77e619250-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:06.785053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.784986    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b6c86e0-9ece-4256-8456-8879c6782878-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:06.786544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.786520    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae77e87-e96d-45d4-a495-7dc77e619250-kube-api-access-dkdfw" pod "bae77e87-e96d-45d4-a495-7dc77e619250" (UID: "bae77e87-e96d-45d4-a495-7dc77e619250"). InnerVolumeSpecName "kube-api-access-dkdfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:06.788126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.788101    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6c86e0-9ece-4256-8456-8879c6782878-kube-api-access-hcfxp" pod "4b6c86e0-9ece-4256-8456-8879c6782878" (UID: "4b6c86e0-9ece-4256-8456-8879c6782878"). InnerVolumeSpecName "kube-api-access-hcfxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:06.883858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.883831    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:38:06.886487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.886462    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcfxp\" (UniqueName: \"kubernetes.io/projected/4b6c86e0-9ece-4256-8456-8879c6782878-kube-api-access-hcfxp\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:06.886612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.886488    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkdfw\" (UniqueName: \"kubernetes.io/projected/bae77e87-e96d-45d4-a495-7dc77e619250-kube-api-access-dkdfw\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:06.987581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.987554    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istiod-ca-cert\") pod \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") "
Nov 03 13:38:06.987709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.987597    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-podinfo\") pod \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") "
Nov 03 13:38:06.987789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.987749    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-token\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-token\") pod \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") "
Nov 03 13:38:06.987844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.987786    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-socket\") pod \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") "
Nov 03 13:38:06.987917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.987901    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-kube-api-access-mhtml\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-kube-api-access-mhtml\") pod \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") "
Nov 03 13:38:06.987957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.987936    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-certs\") pod \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") "
Nov 03 13:38:06.987957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.987953    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-credential-socket\") pod \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") "
Nov 03 13:38:06.988066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.987989    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istiod-ca-cert" pod "cc48ac48-5efb-41c5-971f-a74cd1c3ac23" (UID: "cc48ac48-5efb-41c5-971f-a74cd1c3ac23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:06.988122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988087    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-data\") pod \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") "
Nov 03 13:38:06.988183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988116    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-socket" pod "cc48ac48-5efb-41c5-971f-a74cd1c3ac23" (UID: "cc48ac48-5efb-41c5-971f-a74cd1c3ac23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:06.988183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988128    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-envoy\") pod \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\" (UID: \"cc48ac48-5efb-41c5-971f-a74cd1c3ac23\") "
Nov 03 13:38:06.988416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988263    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-certs" pod "cc48ac48-5efb-41c5-971f-a74cd1c3ac23" (UID: "cc48ac48-5efb-41c5-971f-a74cd1c3ac23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:06.988416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988380    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-credential-socket" pod "cc48ac48-5efb-41c5-971f-a74cd1c3ac23" (UID: "cc48ac48-5efb-41c5-971f-a74cd1c3ac23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:06.988416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988394    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-data" pod "cc48ac48-5efb-41c5-971f-a74cd1c3ac23" (UID: "cc48ac48-5efb-41c5-971f-a74cd1c3ac23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:06.988954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988926    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:06.989076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988957    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:06.989076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988976    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:06.989076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.988990    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:06.989076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.989004    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:06.990094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.990073    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-podinfo" pod "cc48ac48-5efb-41c5-971f-a74cd1c3ac23" (UID: "cc48ac48-5efb-41c5-971f-a74cd1c3ac23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:38:06.990094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.990078    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-token" pod "cc48ac48-5efb-41c5-971f-a74cd1c3ac23" (UID: "cc48ac48-5efb-41c5-971f-a74cd1c3ac23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:06.990255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.990180    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-kube-api-access-mhtml" pod "cc48ac48-5efb-41c5-971f-a74cd1c3ac23" (UID: "cc48ac48-5efb-41c5-971f-a74cd1c3ac23"). InnerVolumeSpecName "kube-api-access-mhtml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:06.990323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:06.990279    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-envoy" pod "cc48ac48-5efb-41c5-971f-a74cd1c3ac23" (UID: "cc48ac48-5efb-41c5-971f-a74cd1c3ac23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:07.090074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.089990    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:07.090074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.090022    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhtml\" (UniqueName: \"kubernetes.io/projected/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-kube-api-access-mhtml\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:07.090074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.090036    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:07.090074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.090048    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc48ac48-5efb-41c5-971f-a74cd1c3ac23-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:07.381754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.381665    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:07.485519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.485483    5420 generic.go:356] "Generic (PLEG): container finished" podID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerID="2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611" exitCode=0
Nov 03 13:38:07.485726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.485533    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"
Nov 03 13:38:07.485726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.485553    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" event={"ID":"cc48ac48-5efb-41c5-971f-a74cd1c3ac23","Type":"ContainerDied","Data":"2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611"}
Nov 03 13:38:07.485726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.485637    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk" event={"ID":"cc48ac48-5efb-41c5-971f-a74cd1c3ac23","Type":"ContainerDied","Data":"b980545192a7a36f514a1f58782a338343e232a4735cbd921fbbb2807a720e48"}
Nov 03 13:38:07.485726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.485666    5420 scope.go:122] "RemoveContainer" containerID="2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611"
Nov 03 13:38:07.487836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.487702    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg" event={"ID":"bae77e87-e96d-45d4-a495-7dc77e619250","Type":"ContainerDied","Data":"24ad0459ac0f5e225698816926ee2adb77c0da35e2da920b7cff97c49547f987"}
Nov 03 13:38:07.487836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.487718    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"
Nov 03 13:38:07.492260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.492199    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h" event={"ID":"4b6c86e0-9ece-4256-8456-8879c6782878","Type":"ContainerDied","Data":"1865d9b06f60127a33967b8d0bd3288ac0a6ba85ba9168f77099502f467640ce"}
Nov 03 13:38:07.492535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.492514    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"
Nov 03 13:38:07.528465 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.528439    5420 scope.go:122] "RemoveContainer" containerID="2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611"
Nov 03 13:38:07.529682 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:38:07.529649    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611\": container with ID starting with 2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611 not found: ID does not exist" containerID="2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611"
Nov 03 13:38:07.529779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.529689    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611"} err="failed to get container status \"2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611\": rpc error: code = NotFound desc = could not find container \"2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611\": container with ID starting with 2c8207c6629d661b2ff7b6c8a929bed70124d96803e15499e061e649eb85d611 not found: ID does not exist"
Nov 03 13:38:07.529779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.529713    5420 scope.go:122] "RemoveContainer" containerID="41196dcba1b59bf70ac9ff3736e59f9745e308dc8d043bb00a1c60ad74123d5f"
Nov 03 13:38:07.537363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.537334    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"]
Nov 03 13:38:07.544808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.544774    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e439d04-588c697959-8dgrg"]
Nov 03 13:38:07.550406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.550387    5420 scope.go:122] "RemoveContainer" containerID="a28d98170c8a25070fa2f596a848dc13dba94eb563b642d97567239d66c185c1"
Nov 03 13:38:07.557099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.557077    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"]
Nov 03 13:38:07.562338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.562317    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-ns6mk"]
Nov 03 13:38:07.585263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.585237    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"]
Nov 03 13:38:07.590855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.590828    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e439d04-6fbd4dcf47-jw96h"]
Nov 03 13:38:07.606444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.606415    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:07.739431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.739397    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:07.902852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:07.898813    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:08.108930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:08.108858    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"]
Nov 03 13:38:08.109079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:08.109028    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy" containerID="cri-o://f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c" gracePeriod=30
Nov 03 13:38:08.250299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:08.250169    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:08.919163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:08.919127    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:09.085360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:09.085330    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6c86e0-9ece-4256-8456-8879c6782878" path="/var/lib/kubelet/pods/4b6c86e0-9ece-4256-8456-8879c6782878/volumes"
Nov 03 13:38:09.085750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:09.085736    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae77e87-e96d-45d4-a495-7dc77e619250" path="/var/lib/kubelet/pods/bae77e87-e96d-45d4-a495-7dc77e619250/volumes"
Nov 03 13:38:09.086193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:09.086172    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" path="/var/lib/kubelet/pods/cc48ac48-5efb-41c5-971f-a74cd1c3ac23/volumes"
Nov 03 13:38:09.590234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:09.582851    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:10.295803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.294644    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:10.302661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.302626    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"]
Nov 03 13:38:10.303493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.303474    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4b6c86e0-9ece-4256-8456-8879c6782878" containerName="payload-pre-processing"
Nov 03 13:38:10.303493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.303492    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6c86e0-9ece-4256-8456-8879c6782878" containerName="payload-pre-processing"
Nov 03 13:38:10.303630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.303507    5420 cpu_manager.go:414] "containerMap: removing container" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy"
Nov 03 13:38:10.303630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.303512    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy"
Nov 03 13:38:10.303630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.303521    5420 cpu_manager.go:414] "containerMap: removing container" podUID="bae77e87-e96d-45d4-a495-7dc77e619250" containerName="payload-processing"
Nov 03 13:38:10.303630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.303526    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae77e87-e96d-45d4-a495-7dc77e619250" containerName="payload-processing"
Nov 03 13:38:10.303761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.303642    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6c86e0-9ece-4256-8456-8879c6782878" containerName="payload-pre-processing"
Nov 03 13:38:10.303761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.303651    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc48ac48-5efb-41c5-971f-a74cd1c3ac23" containerName="istio-proxy"
Nov 03 13:38:10.303761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.303656    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae77e87-e96d-45d4-a495-7dc77e619250" containerName="payload-processing"
Nov 03 13:38:10.309358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.309332    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.313895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.313826    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:38:10.332107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.328576    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:38:10.332107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.328703    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:38:10.332107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.328821    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:38:10.333263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.333121    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"]
Nov 03 13:38:10.339705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.339644    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51afdae-658hl\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.339985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.339959    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51afdae-658hl\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.340332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.340311    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2987p\" (UniqueName: \"kubernetes.io/projected/93b8cc53-7ecc-4292-82e3-6c4eb5226009-kube-api-access-2987p\") pod \"maas-api-revoke-sub-a51afdae-658hl\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.441923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.441895    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2987p\" (UniqueName: \"kubernetes.io/projected/93b8cc53-7ecc-4292-82e3-6c4eb5226009-kube-api-access-2987p\") pod \"maas-api-revoke-sub-a51afdae-658hl\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.442079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.442012    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51afdae-658hl\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.442079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.442069    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51afdae-658hl\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.442734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.442709    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51afdae-658hl\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.444490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.444471    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51afdae-658hl\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.450498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.450475    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2987p\" (UniqueName: \"kubernetes.io/projected/93b8cc53-7ecc-4292-82e3-6c4eb5226009-kube-api-access-2987p\") pod \"maas-api-revoke-sub-a51afdae-658hl\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.654177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.654088    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:10.851908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.851877    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:10.945409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:10.945380    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"]
Nov 03 13:38:11.020377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:11.020339    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:11.529753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:11.529667    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl" event={"ID":"93b8cc53-7ecc-4292-82e3-6c4eb5226009","Type":"ContainerStarted","Data":"1992d15455eb9535b75d99f76fb796213b8d426f1cd979fc652a6e367b41a311"}
Nov 03 13:38:11.529753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:11.529727    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl" event={"ID":"93b8cc53-7ecc-4292-82e3-6c4eb5226009","Type":"ContainerStarted","Data":"60fd866e508b89fbd1a3b5e1a967f04c74a782e3bb0687ef8316822b91d949ed"}
Nov 03 13:38:11.552971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:11.552924    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl" podStartSLOduration=1.552910109 podStartE2EDuration="1.552910109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:11.545960923 +0000 UTC m=+1732.623725950" watchObservedRunningTime="2026-11-03 13:38:11.552910109 +0000 UTC m=+1732.630675134"
Nov 03 13:38:11.700121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:11.700026    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:12.426720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:12.426689    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:12.539177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:12.539143    5420 generic.go:356] "Generic (PLEG): container finished" podID="93b8cc53-7ecc-4292-82e3-6c4eb5226009" containerID="1992d15455eb9535b75d99f76fb796213b8d426f1cd979fc652a6e367b41a311" exitCode=0
Nov 03 13:38:12.539629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:12.539266    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl" event={"ID":"93b8cc53-7ecc-4292-82e3-6c4eb5226009","Type":"ContainerDied","Data":"1992d15455eb9535b75d99f76fb796213b8d426f1cd979fc652a6e367b41a311"}
Nov 03 13:38:13.111008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.110929    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:13.329659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.328537    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:13.415321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.415295    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:38:13.485515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.485479    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-token\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-token\") pod \"7e0c07ab-fa98-462e-8fbc-6119172109cb\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") "
Nov 03 13:38:13.485701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.485533    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-kube-api-access-42js5\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-kube-api-access-42js5\") pod \"7e0c07ab-fa98-462e-8fbc-6119172109cb\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") "
Nov 03 13:38:13.485701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.485640    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-podinfo\") pod \"7e0c07ab-fa98-462e-8fbc-6119172109cb\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") "
Nov 03 13:38:13.485701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.485670    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-certs\") pod \"7e0c07ab-fa98-462e-8fbc-6119172109cb\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") "
Nov 03 13:38:13.485900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.485705    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-envoy\") pod \"7e0c07ab-fa98-462e-8fbc-6119172109cb\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") "
Nov 03 13:38:13.485900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.485862    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-data\") pod \"7e0c07ab-fa98-462e-8fbc-6119172109cb\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") "
Nov 03 13:38:13.486002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.485899    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-socket\") pod \"7e0c07ab-fa98-462e-8fbc-6119172109cb\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") "
Nov 03 13:38:13.486002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.485956    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-credential-socket\") pod \"7e0c07ab-fa98-462e-8fbc-6119172109cb\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") "
Nov 03 13:38:13.486117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486006    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-certs" pod "7e0c07ab-fa98-462e-8fbc-6119172109cb" (UID: "7e0c07ab-fa98-462e-8fbc-6119172109cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:13.486237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486140    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0c07ab-fa98-462e-8fbc-6119172109cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0c07ab-fa98-462e-8fbc-6119172109cb-istiod-ca-cert\") pod \"7e0c07ab-fa98-462e-8fbc-6119172109cb\" (UID: \"7e0c07ab-fa98-462e-8fbc-6119172109cb\") "
Nov 03 13:38:13.486237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486174    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-data" pod "7e0c07ab-fa98-462e-8fbc-6119172109cb" (UID: "7e0c07ab-fa98-462e-8fbc-6119172109cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:13.486237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486191    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-socket" pod "7e0c07ab-fa98-462e-8fbc-6119172109cb" (UID: "7e0c07ab-fa98-462e-8fbc-6119172109cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:13.486453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486418    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-credential-socket" pod "7e0c07ab-fa98-462e-8fbc-6119172109cb" (UID: "7e0c07ab-fa98-462e-8fbc-6119172109cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:13.486737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486710    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0c07ab-fa98-462e-8fbc-6119172109cb-istiod-ca-cert" pod "7e0c07ab-fa98-462e-8fbc-6119172109cb" (UID: "7e0c07ab-fa98-462e-8fbc-6119172109cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:13.486953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486930    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0c07ab-fa98-462e-8fbc-6119172109cb-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.487026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486959    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.487026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486974    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.487026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.486988    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.487026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.487005    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.489128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.489098    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-envoy" pod "7e0c07ab-fa98-462e-8fbc-6119172109cb" (UID: "7e0c07ab-fa98-462e-8fbc-6119172109cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:13.489275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.489244    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-kube-api-access-42js5" pod "7e0c07ab-fa98-462e-8fbc-6119172109cb" (UID: "7e0c07ab-fa98-462e-8fbc-6119172109cb"). InnerVolumeSpecName "kube-api-access-42js5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:13.489410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.489387    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-podinfo" pod "7e0c07ab-fa98-462e-8fbc-6119172109cb" (UID: "7e0c07ab-fa98-462e-8fbc-6119172109cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:38:13.489458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.489429    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-token" pod "7e0c07ab-fa98-462e-8fbc-6119172109cb" (UID: "7e0c07ab-fa98-462e-8fbc-6119172109cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:13.556757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.556713    5420 generic.go:356] "Generic (PLEG): container finished" podID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerID="f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c" exitCode=0
Nov 03 13:38:13.557266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.557118    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"
Nov 03 13:38:13.557722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.557658    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" event={"ID":"7e0c07ab-fa98-462e-8fbc-6119172109cb","Type":"ContainerDied","Data":"f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c"}
Nov 03 13:38:13.557722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.557721    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq" event={"ID":"7e0c07ab-fa98-462e-8fbc-6119172109cb","Type":"ContainerDied","Data":"c5bd191c19de735d643cc8a00f1ffb415e6871bc3486634833995c708e8af45e"}
Nov 03 13:38:13.557991 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.557750    5420 scope.go:122] "RemoveContainer" containerID="f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c"
Nov 03 13:38:13.574054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.574033    5420 scope.go:122] "RemoveContainer" containerID="f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c"
Nov 03 13:38:13.574426 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:38:13.574401    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c\": container with ID starting with f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c not found: ID does not exist" containerID="f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c"
Nov 03 13:38:13.574523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.574432    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c"} err="failed to get container status \"f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c\": rpc error: code = NotFound desc = could not find container \"f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c\": container with ID starting with f1f94dfc63c277a25d0c13629ad5869de6796df67b6e4803486932e0dec1335c not found: ID does not exist"
Nov 03 13:38:13.586781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.586755    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"]
Nov 03 13:38:13.588357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.588333    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.588431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.588363    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42js5\" (UniqueName: \"kubernetes.io/projected/7e0c07ab-fa98-462e-8fbc-6119172109cb-kube-api-access-42js5\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.588431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.588379    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.588431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.588392    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0c07ab-fa98-462e-8fbc-6119172109cb-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.593317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.593296    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rwkwq"]
Nov 03 13:38:13.690575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.690552    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:13.790773 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.790740    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-openshift-ca\") pod \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") "
Nov 03 13:38:13.790947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.790782    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-xks-ca\") pod \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") "
Nov 03 13:38:13.791087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.791064    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-openshift-ca" pod "93b8cc53-7ecc-4292-82e3-6c4eb5226009" (UID: "93b8cc53-7ecc-4292-82e3-6c4eb5226009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:13.791163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.791142    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b8cc53-7ecc-4292-82e3-6c4eb5226009-kube-api-access-2987p\" (UniqueName: \"kubernetes.io/projected/93b8cc53-7ecc-4292-82e3-6c4eb5226009-kube-api-access-2987p\") pod \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\" (UID: \"93b8cc53-7ecc-4292-82e3-6c4eb5226009\") "
Nov 03 13:38:13.791733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.791713    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.792946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.792925    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-xks-ca" pod "93b8cc53-7ecc-4292-82e3-6c4eb5226009" (UID: "93b8cc53-7ecc-4292-82e3-6c4eb5226009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:38:13.793202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.793184    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b8cc53-7ecc-4292-82e3-6c4eb5226009-kube-api-access-2987p" pod "93b8cc53-7ecc-4292-82e3-6c4eb5226009" (UID: "93b8cc53-7ecc-4292-82e3-6c4eb5226009"). InnerVolumeSpecName "kube-api-access-2987p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:13.893178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.893144    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2987p\" (UniqueName: \"kubernetes.io/projected/93b8cc53-7ecc-4292-82e3-6c4eb5226009-kube-api-access-2987p\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.893178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.893173    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b8cc53-7ecc-4292-82e3-6c4eb5226009-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:13.931731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:13.929128    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:14.566960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:14.566932    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"
Nov 03 13:38:14.566960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:14.566943    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl" event={"ID":"93b8cc53-7ecc-4292-82e3-6c4eb5226009","Type":"ContainerDied","Data":"60fd866e508b89fbd1a3b5e1a967f04c74a782e3bb0687ef8316822b91d949ed"}
Nov 03 13:38:14.567530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:14.567030    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60fd866e508b89fbd1a3b5e1a967f04c74a782e3bb0687ef8316822b91d949ed"
Nov 03 13:38:14.594429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:14.593354    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:15.087162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:15.087131    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" path="/var/lib/kubelet/pods/7e0c07ab-fa98-462e-8fbc-6119172109cb/volumes"
Nov 03 13:38:15.273398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:15.273368    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:15.569235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:15.565643    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:16.234244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:16.229784    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:16.822861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:16.822829    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:16.984823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:16.984793    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:17.264955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:17.264917    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:17.495131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:17.495096    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:38:17.495356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:17.495175    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:38:18.007443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:18.007398    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:18.706541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:18.706496    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:19.214563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:19.214526    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"]
Nov 03 13:38:19.214976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:19.214722    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy" containerID="cri-o://af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0" gracePeriod=30
Nov 03 13:38:19.218165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:19.218134    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:19.445759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:19.445668    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:20.328002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.327966    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:20.535013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.534978    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"]
Nov 03 13:38:20.535709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.535694    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy"
Nov 03 13:38:20.535775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.535710    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy"
Nov 03 13:38:20.535775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.535739    5420 cpu_manager.go:414] "containerMap: removing container" podUID="93b8cc53-7ecc-4292-82e3-6c4eb5226009" containerName="revoke-keys"
Nov 03 13:38:20.535775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.535744    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b8cc53-7ecc-4292-82e3-6c4eb5226009" containerName="revoke-keys"
Nov 03 13:38:20.535883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.535858    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0c07ab-fa98-462e-8fbc-6119172109cb" containerName="istio-proxy"
Nov 03 13:38:20.535883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.535871    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b8cc53-7ecc-4292-82e3-6c4eb5226009" containerName="revoke-keys"
Nov 03 13:38:20.540203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.540180    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.545676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.545646    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:38:20.545676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.545655    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:38:20.545676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.545677    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:38:20.545891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.545696    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:38:20.547616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.547587    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"]
Nov 03 13:38:20.634318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.634201    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvw4\" (UniqueName: \"kubernetes.io/projected/b6b1ad37-709c-403d-8a4b-27712f6eb522-kube-api-access-qxvw4\") pod \"maas-api-revoke-sub-08ecc2ff-8k9rb\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.634476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.634461    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ecc2ff-8k9rb\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.634521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.634507    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ecc2ff-8k9rb\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.736066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.736031    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvw4\" (UniqueName: \"kubernetes.io/projected/b6b1ad37-709c-403d-8a4b-27712f6eb522-kube-api-access-qxvw4\") pod \"maas-api-revoke-sub-08ecc2ff-8k9rb\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.736298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.736206    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ecc2ff-8k9rb\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.736298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.736274    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ecc2ff-8k9rb\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.736886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.736865    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ecc2ff-8k9rb\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.738662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.738640    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ecc2ff-8k9rb\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.745770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.745746    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvw4\" (UniqueName: \"kubernetes.io/projected/b6b1ad37-709c-403d-8a4b-27712f6eb522-kube-api-access-qxvw4\") pod \"maas-api-revoke-sub-08ecc2ff-8k9rb\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.858619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.858583    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:20.956355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:20.956322    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:21.111056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:21.111017    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:21.155551 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:38:21.154978    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b1ad37_709c_403d_8a4b_27712f6eb522.slice/crio-dfabc836699f354c591d2690b937d83ed58f935559040b718d233ca7b2232a4c WatchSource:0}: Error finding container dfabc836699f354c591d2690b937d83ed58f935559040b718d233ca7b2232a4c: Status 404 returned error can't find the container with id dfabc836699f354c591d2690b937d83ed58f935559040b718d233ca7b2232a4c
Nov 03 13:38:21.155884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:21.155862    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"]
Nov 03 13:38:21.472856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:21.472733    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:21.634847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:21.634815    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb" event={"ID":"b6b1ad37-709c-403d-8a4b-27712f6eb522","Type":"ContainerStarted","Data":"82ae1f762e8e01e5f52a85188b44515ce9162b3be487469d539400ccf2e71ca2"}
Nov 03 13:38:21.635027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:21.634868    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb" event={"ID":"b6b1ad37-709c-403d-8a4b-27712f6eb522","Type":"ContainerStarted","Data":"dfabc836699f354c591d2690b937d83ed58f935559040b718d233ca7b2232a4c"}
Nov 03 13:38:21.685306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:21.685201    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb" podStartSLOduration=1.685188959 podStartE2EDuration="1.685188959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:21.681124887 +0000 UTC m=+1742.758889917" watchObservedRunningTime="2026-11-03 13:38:21.685188959 +0000 UTC m=+1742.762953985"
Nov 03 13:38:21.919800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:21.919763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:22.016202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:22.013455    5420 scope.go:122] "RemoveContainer" containerID="d3115068baf650ee554ce4b46af63a38ef57a5f2c2427f9bffd441d98d3df330"
Nov 03 13:38:22.083068 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:22.082930    5420 scope.go:122] "RemoveContainer" containerID="308d8b53f17d38499c5c86167824a4936dd3d7f55ae105fb481d7bf32935c602"
Nov 03 13:38:22.114245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:22.109622    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:22.155172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:22.155136    5420 scope.go:122] "RemoveContainer" containerID="45d44b2b8ab985b90280afb62ddbffccb7d14a2b05168941da23612fce703189"
Nov 03 13:38:22.215398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:22.215374    5420 scope.go:122] "RemoveContainer" containerID="85f30a2265c23e9342d667a51955af6454c3dcacfe33b05835172a39bc42d5bb"
Nov 03 13:38:22.226010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:22.225985    5420 scope.go:122] "RemoveContainer" containerID="aacbfc4e651dd7170f67e16d753af63d15e9b12d4839d1dd70acf178aebd01eb"
Nov 03 13:38:22.357717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:22.357678    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:22.645033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:22.644949    5420 generic.go:356] "Generic (PLEG): container finished" podID="b6b1ad37-709c-403d-8a4b-27712f6eb522" containerID="82ae1f762e8e01e5f52a85188b44515ce9162b3be487469d539400ccf2e71ca2" exitCode=0
Nov 03 13:38:22.645455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:22.645035    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb" event={"ID":"b6b1ad37-709c-403d-8a4b-27712f6eb522","Type":"ContainerDied","Data":"82ae1f762e8e01e5f52a85188b44515ce9162b3be487469d539400ccf2e71ca2"}
Nov 03 13:38:23.826007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:23.825980    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:23.867163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:23.867122    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:24.029763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.029723    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b1ad37-709c-403d-8a4b-27712f6eb522-kube-api-access-qxvw4\" (UniqueName: \"kubernetes.io/projected/b6b1ad37-709c-403d-8a4b-27712f6eb522-kube-api-access-qxvw4\") pod \"b6b1ad37-709c-403d-8a4b-27712f6eb522\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") "
Nov 03 13:38:24.029983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.029801    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-xks-ca\") pod \"b6b1ad37-709c-403d-8a4b-27712f6eb522\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") "
Nov 03 13:38:24.029983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.029938    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-openshift-ca\") pod \"b6b1ad37-709c-403d-8a4b-27712f6eb522\" (UID: \"b6b1ad37-709c-403d-8a4b-27712f6eb522\") "
Nov 03 13:38:24.030365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.030331    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-openshift-ca" pod "b6b1ad37-709c-403d-8a4b-27712f6eb522" (UID: "b6b1ad37-709c-403d-8a4b-27712f6eb522"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:24.030775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.030748    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.032066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.032039    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b1ad37-709c-403d-8a4b-27712f6eb522-kube-api-access-qxvw4" pod "b6b1ad37-709c-403d-8a4b-27712f6eb522" (UID: "b6b1ad37-709c-403d-8a4b-27712f6eb522"). InnerVolumeSpecName "kube-api-access-qxvw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:24.032066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.032043    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-xks-ca" pod "b6b1ad37-709c-403d-8a4b-27712f6eb522" (UID: "b6b1ad37-709c-403d-8a4b-27712f6eb522"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:38:24.131986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.131953    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1ad37-709c-403d-8a4b-27712f6eb522-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.131986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.131981    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvw4\" (UniqueName: \"kubernetes.io/projected/b6b1ad37-709c-403d-8a4b-27712f6eb522-kube-api-access-qxvw4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.472100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.472069    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:38:24.537327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537290    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-data\") pod \"8069b68c-2d60-473b-968c-9013630f3a00\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") "
Nov 03 13:38:24.537518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537420    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8069b68c-2d60-473b-968c-9013630f3a00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8069b68c-2d60-473b-968c-9013630f3a00-istio-podinfo\") pod \"8069b68c-2d60-473b-968c-9013630f3a00\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") "
Nov 03 13:38:24.537518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537488    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-kube-api-access-j94lg\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-kube-api-access-j94lg\") pod \"8069b68c-2d60-473b-968c-9013630f3a00\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") "
Nov 03 13:38:24.537648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537541    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-credential-socket\") pod \"8069b68c-2d60-473b-968c-9013630f3a00\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") "
Nov 03 13:38:24.537648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537588    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-data" pod "8069b68c-2d60-473b-968c-9013630f3a00" (UID: "8069b68c-2d60-473b-968c-9013630f3a00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:24.537648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537602    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8069b68c-2d60-473b-968c-9013630f3a00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8069b68c-2d60-473b-968c-9013630f3a00-istiod-ca-cert\") pod \"8069b68c-2d60-473b-968c-9013630f3a00\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") "
Nov 03 13:38:24.537806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537692    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-socket\") pod \"8069b68c-2d60-473b-968c-9013630f3a00\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") "
Nov 03 13:38:24.537806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537767    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-certs\") pod \"8069b68c-2d60-473b-968c-9013630f3a00\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") "
Nov 03 13:38:24.537928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537858    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-istio-token\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-istio-token\") pod \"8069b68c-2d60-473b-968c-9013630f3a00\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") "
Nov 03 13:38:24.537996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537939    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-envoy\") pod \"8069b68c-2d60-473b-968c-9013630f3a00\" (UID: \"8069b68c-2d60-473b-968c-9013630f3a00\") "
Nov 03 13:38:24.538058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.537985    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8069b68c-2d60-473b-968c-9013630f3a00-istiod-ca-cert" pod "8069b68c-2d60-473b-968c-9013630f3a00" (UID: "8069b68c-2d60-473b-968c-9013630f3a00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:24.538225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.538187    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-socket" pod "8069b68c-2d60-473b-968c-9013630f3a00" (UID: "8069b68c-2d60-473b-968c-9013630f3a00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:24.538358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.538333    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-credential-socket" pod "8069b68c-2d60-473b-968c-9013630f3a00" (UID: "8069b68c-2d60-473b-968c-9013630f3a00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:24.538760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.538731    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.538864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.538762    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.538864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.538779    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8069b68c-2d60-473b-968c-9013630f3a00-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.538864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.538795    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.539092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.539002    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-certs" pod "8069b68c-2d60-473b-968c-9013630f3a00" (UID: "8069b68c-2d60-473b-968c-9013630f3a00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:24.540518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.540491    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8069b68c-2d60-473b-968c-9013630f3a00-istio-podinfo" pod "8069b68c-2d60-473b-968c-9013630f3a00" (UID: "8069b68c-2d60-473b-968c-9013630f3a00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:38:24.541352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.541324    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-istio-token" pod "8069b68c-2d60-473b-968c-9013630f3a00" (UID: "8069b68c-2d60-473b-968c-9013630f3a00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:24.542634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.542610    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-envoy" pod "8069b68c-2d60-473b-968c-9013630f3a00" (UID: "8069b68c-2d60-473b-968c-9013630f3a00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:38:24.543794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.543771    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-kube-api-access-j94lg" pod "8069b68c-2d60-473b-968c-9013630f3a00" (UID: "8069b68c-2d60-473b-968c-9013630f3a00"). InnerVolumeSpecName "kube-api-access-j94lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:24.595660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.595598    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:24.640479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.640440    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8069b68c-2d60-473b-968c-9013630f3a00-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.640479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.640475    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j94lg\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-kube-api-access-j94lg\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.640800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.640493    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.640800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.640506    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8069b68c-2d60-473b-968c-9013630f3a00-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.640800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.640518    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8069b68c-2d60-473b-968c-9013630f3a00-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:24.670482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.670451    5420 generic.go:356] "Generic (PLEG): container finished" podID="8069b68c-2d60-473b-968c-9013630f3a00" containerID="af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0" exitCode=0
Nov 03 13:38:24.670717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.670698    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" event={"ID":"8069b68c-2d60-473b-968c-9013630f3a00","Type":"ContainerDied","Data":"af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0"}
Nov 03 13:38:24.670800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.670754    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78" event={"ID":"8069b68c-2d60-473b-968c-9013630f3a00","Type":"ContainerDied","Data":"68b8249689450c731c21310b08cab5d2f18e8228908dbc4513c1ca285218ddbf"}
Nov 03 13:38:24.670800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.670787    5420 scope.go:122] "RemoveContainer" containerID="af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0"
Nov 03 13:38:24.670970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.670955    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"
Nov 03 13:38:24.673576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.673537    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb" event={"ID":"b6b1ad37-709c-403d-8a4b-27712f6eb522","Type":"ContainerDied","Data":"dfabc836699f354c591d2690b937d83ed58f935559040b718d233ca7b2232a4c"}
Nov 03 13:38:24.673701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.673592    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfabc836699f354c591d2690b937d83ed58f935559040b718d233ca7b2232a4c"
Nov 03 13:38:24.673701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.673624    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"
Nov 03 13:38:24.686227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.686190    5420 scope.go:122] "RemoveContainer" containerID="af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0"
Nov 03 13:38:24.686580 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:38:24.686560    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0\": container with ID starting with af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0 not found: ID does not exist" containerID="af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0"
Nov 03 13:38:24.686635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.686593    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0"} err="failed to get container status \"af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0\": rpc error: code = NotFound desc = could not find container \"af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0\": container with ID starting with af7bc83ad07c5622b9fec52ec3ad899a38fa6814f1c88d23ccc45b778a9591e0 not found: ID does not exist"
Nov 03 13:38:24.704197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.704154    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"]
Nov 03 13:38:24.712670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:24.712645    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-6bp78"]
Nov 03 13:38:24.734964 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:38:24.734927    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:38:25.085942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:25.085912    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8069b68c-2d60-473b-968c-9013630f3a00" path="/var/lib/kubelet/pods/8069b68c-2d60-473b-968c-9013630f3a00/volumes"
Nov 03 13:38:25.301278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:25.301202    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:25.960237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:25.957113    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:26.698754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.698724    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"]
Nov 03 13:38:26.699451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.699435    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b6b1ad37-709c-403d-8a4b-27712f6eb522" containerName="revoke-keys"
Nov 03 13:38:26.699451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.699451    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b1ad37-709c-403d-8a4b-27712f6eb522" containerName="revoke-keys"
Nov 03 13:38:26.699546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.699475    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy"
Nov 03 13:38:26.699546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.699481    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy"
Nov 03 13:38:26.699611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.699582    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b1ad37-709c-403d-8a4b-27712f6eb522" containerName="revoke-keys"
Nov 03 13:38:26.699611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.699594    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8069b68c-2d60-473b-968c-9013630f3a00" containerName="istio-proxy"
Nov 03 13:38:26.705198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.705177    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.712623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.712599    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"]
Nov 03 13:38:26.714397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.714373    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9add72\"/\"rate-test-model-9add72-kserve-self-signed-certs\""
Nov 03 13:38:26.714515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.714411    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9add72\"/\"default-dockercfg-4k9xb\""
Nov 03 13:38:26.714620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.714543    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9add72\"/\"kube-root-ca.crt\""
Nov 03 13:38:26.714750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.714706    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9add72\"/\"openshift-service-ca.crt\""
Nov 03 13:38:26.719955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.719932    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:26.773304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.773274    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-home\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.773481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.773312    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-model-cache\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.773481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.773334    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-dshm\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.773481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.773397    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c82ef7-9eee-408d-8d33-0fd166280fba-tls-certs\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.773672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.773515    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65vx\" (UniqueName: \"kubernetes.io/projected/e1c82ef7-9eee-408d-8d33-0fd166280fba-kube-api-access-m65vx\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.773750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.773733    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-tmp-dir\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.870182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.870149    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:26.877259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.875380    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-tmp-dir\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.877259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.875525    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-home\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.877259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.875567    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-model-cache\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.877259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.875596    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-dshm\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.877259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.875629    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c82ef7-9eee-408d-8d33-0fd166280fba-tls-certs\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.877259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.875692    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65vx\" (UniqueName: \"kubernetes.io/projected/e1c82ef7-9eee-408d-8d33-0fd166280fba-kube-api-access-m65vx\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.877259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.876592    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-tmp-dir\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.877259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.876845    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-home\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.877259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.877074    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-model-cache\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.907144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.907117    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c82ef7-9eee-408d-8d33-0fd166280fba-tls-certs\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.908038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.908004    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65vx\" (UniqueName: \"kubernetes.io/projected/e1c82ef7-9eee-408d-8d33-0fd166280fba-kube-api-access-m65vx\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:26.911556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:26.908535    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-dshm\") pod \"rate-test-model-9add72-kserve-55486764f-2fgwx\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") " pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:27.023671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:27.023636    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:27.327828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:27.327798    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"]
Nov 03 13:38:27.527939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:27.526669    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:27.707001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:27.706956    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx" event={"ID":"e1c82ef7-9eee-408d-8d33-0fd166280fba","Type":"ContainerStarted","Data":"46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944"}
Nov 03 13:38:27.707570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:27.707018    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx" event={"ID":"e1c82ef7-9eee-408d-8d33-0fd166280fba","Type":"ContainerStarted","Data":"8cae73f84ee64dad85e7cceb9b5d73d9fce190e63cebf6e543ad3289e86e9d59"}
Nov 03 13:38:27.732745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:27.732695    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx" podStartSLOduration=1.732681339 podStartE2EDuration="1.732681339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:27.731538544 +0000 UTC m=+1748.809303571" watchObservedRunningTime="2026-11-03 13:38:27.732681339 +0000 UTC m=+1748.810446402"
Nov 03 13:38:28.024098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:28.023990    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:28.419282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:28.413071    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:29.029412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:29.028028    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:29.500226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:29.500176    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:29.691509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:29.691477    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:29.857287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:29.852544    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:30.008591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:30.008542    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:30.546059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:30.546027    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:31.186059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.186029    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"]
Nov 03 13:38:31.190620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.190604    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.197772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.197737    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:38:31.198154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.198134    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:38:31.198396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.198379    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:38:31.198522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.198382    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:38:31.200071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.200050    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"]
Nov 03 13:38:31.262615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.262574    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8469b741-w2hl6\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.263490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.263461    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8469b741-w2hl6\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.263686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.263665    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2ws\" (UniqueName: \"kubernetes.io/projected/f5039a0c-e0a7-4520-b051-f50048f1a663-kube-api-access-ld2ws\") pod \"maas-api-revoke-sub-8469b741-w2hl6\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.272603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.272545    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:31.365872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.365841    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8469b741-w2hl6\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.366090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.365885    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2ws\" (UniqueName: \"kubernetes.io/projected/f5039a0c-e0a7-4520-b051-f50048f1a663-kube-api-access-ld2ws\") pod \"maas-api-revoke-sub-8469b741-w2hl6\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.366090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.366046    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8469b741-w2hl6\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.366609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.366589    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8469b741-w2hl6\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.368421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.368399    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8469b741-w2hl6\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.374693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.374669    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2ws\" (UniqueName: \"kubernetes.io/projected/f5039a0c-e0a7-4520-b051-f50048f1a663-kube-api-access-ld2ws\") pod \"maas-api-revoke-sub-8469b741-w2hl6\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.508877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.508846    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:31.804883 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:38:31.804849    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5039a0c_e0a7_4520_b051_f50048f1a663.slice/crio-997c1c1232a0bf2253f69c1fd64011f448cfe698e45cece7e31f5f84d2aff251 WatchSource:0}: Error finding container 997c1c1232a0bf2253f69c1fd64011f448cfe698e45cece7e31f5f84d2aff251: Status 404 returned error can't find the container with id 997c1c1232a0bf2253f69c1fd64011f448cfe698e45cece7e31f5f84d2aff251
Nov 03 13:38:31.805400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:31.805053    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"]
Nov 03 13:38:32.194739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:32.188594    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:32.748099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:32.748067    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6" event={"ID":"f5039a0c-e0a7-4520-b051-f50048f1a663","Type":"ContainerStarted","Data":"8eb7aa7ed7930fa79728ca6431034b3975d219f5f9b029f8ad91b77bf1e01048"}
Nov 03 13:38:32.748314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:32.748131    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6" event={"ID":"f5039a0c-e0a7-4520-b051-f50048f1a663","Type":"ContainerStarted","Data":"997c1c1232a0bf2253f69c1fd64011f448cfe698e45cece7e31f5f84d2aff251"}
Nov 03 13:38:32.772087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:32.772020    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6" podStartSLOduration=1.7719887760000002 podStartE2EDuration="1.771988776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:32.764581283 +0000 UTC m=+1753.842346309" watchObservedRunningTime="2026-11-03 13:38:32.771988776 +0000 UTC m=+1753.849753802"
Nov 03 13:38:32.906325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:32.906257    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:33.508754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:33.508685    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:33.699248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:33.699141    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:33.761528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:33.761437    5420 generic.go:356] "Generic (PLEG): container finished" podID="f5039a0c-e0a7-4520-b051-f50048f1a663" containerID="8eb7aa7ed7930fa79728ca6431034b3975d219f5f9b029f8ad91b77bf1e01048" exitCode=0
Nov 03 13:38:33.761712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:33.761611    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6" event={"ID":"f5039a0c-e0a7-4520-b051-f50048f1a663","Type":"ContainerDied","Data":"8eb7aa7ed7930fa79728ca6431034b3975d219f5f9b029f8ad91b77bf1e01048"}
Nov 03 13:38:34.683861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:34.683385    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:34.942813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:34.942756    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:35.125236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.121574    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-xks-ca\") pod \"f5039a0c-e0a7-4520-b051-f50048f1a663\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") "
Nov 03 13:38:35.125236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.121832    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-openshift-ca\") pod \"f5039a0c-e0a7-4520-b051-f50048f1a663\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") "
Nov 03 13:38:35.125236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.121960    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5039a0c-e0a7-4520-b051-f50048f1a663-kube-api-access-ld2ws\" (UniqueName: \"kubernetes.io/projected/f5039a0c-e0a7-4520-b051-f50048f1a663-kube-api-access-ld2ws\") pod \"f5039a0c-e0a7-4520-b051-f50048f1a663\" (UID: \"f5039a0c-e0a7-4520-b051-f50048f1a663\") "
Nov 03 13:38:35.125236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.123707    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-openshift-ca" pod "f5039a0c-e0a7-4520-b051-f50048f1a663" (UID: "f5039a0c-e0a7-4520-b051-f50048f1a663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:35.137236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.127575    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-xks-ca" pod "f5039a0c-e0a7-4520-b051-f50048f1a663" (UID: "f5039a0c-e0a7-4520-b051-f50048f1a663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:38:35.137236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.130144    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5039a0c-e0a7-4520-b051-f50048f1a663-kube-api-access-ld2ws" pod "f5039a0c-e0a7-4520-b051-f50048f1a663" (UID: "f5039a0c-e0a7-4520-b051-f50048f1a663"). InnerVolumeSpecName "kube-api-access-ld2ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:35.223996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.223951    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.223996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.223995    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5039a0c-e0a7-4520-b051-f50048f1a663-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.224250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.224020    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld2ws\" (UniqueName: \"kubernetes.io/projected/f5039a0c-e0a7-4520-b051-f50048f1a663-kube-api-access-ld2ws\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:35.781784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.781756    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"
Nov 03 13:38:35.782291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.781793    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6" event={"ID":"f5039a0c-e0a7-4520-b051-f50048f1a663","Type":"ContainerDied","Data":"997c1c1232a0bf2253f69c1fd64011f448cfe698e45cece7e31f5f84d2aff251"}
Nov 03 13:38:35.782291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.781856    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="997c1c1232a0bf2253f69c1fd64011f448cfe698e45cece7e31f5f84d2aff251"
Nov 03 13:38:35.988241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:35.987963    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:36.870705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:36.870671    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:37.023999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:37.023966    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:37.036587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:37.036561    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:37.383730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:37.383697    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:37.809789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:37.809757    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:38:37.815929 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:37.815902    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:37.961111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:37.961075    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:38.148839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.148756    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:38.350752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.350722    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"]
Nov 03 13:38:38.351444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.351423    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f5039a0c-e0a7-4520-b051-f50048f1a663" containerName="revoke-keys"
Nov 03 13:38:38.351522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.351452    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5039a0c-e0a7-4520-b051-f50048f1a663" containerName="revoke-keys"
Nov 03 13:38:38.351593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.351584    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5039a0c-e0a7-4520-b051-f50048f1a663" containerName="revoke-keys"
Nov 03 13:38:38.355306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.355288    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.364501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.364479    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:38:38.364501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.364481    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:38:38.368357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.368329    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:38:38.368478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.368373    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:38:38.374307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.374281    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"]
Nov 03 13:38:38.503823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.503794    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c8e5fc-stf7k\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.504006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.503829    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c8e5fc-stf7k\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.504006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.503924    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d85lz\" (UniqueName: \"kubernetes.io/projected/78877620-c79b-47d5-b64d-23ea310be7c2-kube-api-access-d85lz\") pod \"maas-api-revoke-sub-b2c8e5fc-stf7k\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.604986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.604952    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d85lz\" (UniqueName: \"kubernetes.io/projected/78877620-c79b-47d5-b64d-23ea310be7c2-kube-api-access-d85lz\") pod \"maas-api-revoke-sub-b2c8e5fc-stf7k\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.605173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.605161    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c8e5fc-stf7k\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.605258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.605183    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c8e5fc-stf7k\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.605870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.605846    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c8e5fc-stf7k\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.607513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.607489    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c8e5fc-stf7k\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.613896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.613868    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d85lz\" (UniqueName: \"kubernetes.io/projected/78877620-c79b-47d5-b64d-23ea310be7c2-kube-api-access-d85lz\") pod \"maas-api-revoke-sub-b2c8e5fc-stf7k\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.670340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.670315    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:38.969260 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:38:38.957031    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78877620_c79b_47d5_b64d_23ea310be7c2.slice/crio-2bd3c13814950d0b1673e51226c3d7c2dc5d1468439cc2c60e5fd900e6add44b WatchSource:0}: Error finding container 2bd3c13814950d0b1673e51226c3d7c2dc5d1468439cc2c60e5fd900e6add44b: Status 404 returned error can't find the container with id 2bd3c13814950d0b1673e51226c3d7c2dc5d1468439cc2c60e5fd900e6add44b
Nov 03 13:38:38.969260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:38.957902    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"]
Nov 03 13:38:39.138133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:39.138082    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:39.816595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:39.816561    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k" event={"ID":"78877620-c79b-47d5-b64d-23ea310be7c2","Type":"ContainerStarted","Data":"d96a558da369378be0f9e68cb432b1d082269f4a02c6976b05970d7ecd398b17"}
Nov 03 13:38:39.816784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:39.816622    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k" event={"ID":"78877620-c79b-47d5-b64d-23ea310be7c2","Type":"ContainerStarted","Data":"2bd3c13814950d0b1673e51226c3d7c2dc5d1468439cc2c60e5fd900e6add44b"}
Nov 03 13:38:39.841566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:39.841517    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k" podStartSLOduration=1.841505285 podStartE2EDuration="1.841505285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:39.836592236 +0000 UTC m=+1760.914357262" watchObservedRunningTime="2026-11-03 13:38:39.841505285 +0000 UTC m=+1760.919270310"
Nov 03 13:38:40.190723 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:40.190634    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:40.827231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:40.827179    5420 generic.go:356] "Generic (PLEG): container finished" podID="78877620-c79b-47d5-b64d-23ea310be7c2" containerID="d96a558da369378be0f9e68cb432b1d082269f4a02c6976b05970d7ecd398b17" exitCode=0
Nov 03 13:38:40.827435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:40.827246    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k" event={"ID":"78877620-c79b-47d5-b64d-23ea310be7c2","Type":"ContainerDied","Data":"d96a558da369378be0f9e68cb432b1d082269f4a02c6976b05970d7ecd398b17"}
Nov 03 13:38:40.968235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:40.968036    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:41.571367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.571330    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"]
Nov 03 13:38:41.577352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.577324    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.583653 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.583622    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"]
Nov 03 13:38:41.652554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.652517    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbhbs\" (UniqueName: \"kubernetes.io/projected/cfd0dba9-afd5-42af-8aac-5c691917fd4f-kube-api-access-mbhbs\") pod \"maas-api-revoke-sub-cbd29c96-z9hzr\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.652748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.652654    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd29c96-z9hzr\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.652841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.652824    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd29c96-z9hzr\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.733720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.725835    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:41.754912 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.754862    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd29c96-z9hzr\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.755139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.755119    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbhbs\" (UniqueName: \"kubernetes.io/projected/cfd0dba9-afd5-42af-8aac-5c691917fd4f-kube-api-access-mbhbs\") pod \"maas-api-revoke-sub-cbd29c96-z9hzr\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.755477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.755177    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd29c96-z9hzr\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.755957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.755936    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd29c96-z9hzr\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.757872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.757848    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd29c96-z9hzr\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.771917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.771890    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbhbs\" (UniqueName: \"kubernetes.io/projected/cfd0dba9-afd5-42af-8aac-5c691917fd4f-kube-api-access-mbhbs\") pod \"maas-api-revoke-sub-cbd29c96-z9hzr\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:41.900769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:41.900263    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:42.100229 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.100185    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:42.162142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.162055    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78877620-c79b-47d5-b64d-23ea310be7c2-kube-api-access-d85lz\" (UniqueName: \"kubernetes.io/projected/78877620-c79b-47d5-b64d-23ea310be7c2-kube-api-access-d85lz\") pod \"78877620-c79b-47d5-b64d-23ea310be7c2\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") "
Nov 03 13:38:42.162339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.162199    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-xks-ca\") pod \"78877620-c79b-47d5-b64d-23ea310be7c2\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") "
Nov 03 13:38:42.162416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.162328    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-openshift-ca\") pod \"78877620-c79b-47d5-b64d-23ea310be7c2\" (UID: \"78877620-c79b-47d5-b64d-23ea310be7c2\") "
Nov 03 13:38:42.162774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.162745    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-openshift-ca" pod "78877620-c79b-47d5-b64d-23ea310be7c2" (UID: "78877620-c79b-47d5-b64d-23ea310be7c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:42.163030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.162992    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:42.164557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.164529    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-xks-ca" pod "78877620-c79b-47d5-b64d-23ea310be7c2" (UID: "78877620-c79b-47d5-b64d-23ea310be7c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:38:42.164892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.164865    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78877620-c79b-47d5-b64d-23ea310be7c2-kube-api-access-d85lz" pod "78877620-c79b-47d5-b64d-23ea310be7c2" (UID: "78877620-c79b-47d5-b64d-23ea310be7c2"). InnerVolumeSpecName "kube-api-access-d85lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:42.212450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.212419    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"]
Nov 03 13:38:42.212573 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:38:42.212515    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd0dba9_afd5_42af_8aac_5c691917fd4f.slice/crio-3ba4c0384d93ba379dca4cff840b919425de737ef96342e5a9d85a7e5020ab31 WatchSource:0}: Error finding container 3ba4c0384d93ba379dca4cff840b919425de737ef96342e5a9d85a7e5020ab31: Status 404 returned error can't find the container with id 3ba4c0384d93ba379dca4cff840b919425de737ef96342e5a9d85a7e5020ab31
Nov 03 13:38:42.265014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.264987    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78877620-c79b-47d5-b64d-23ea310be7c2-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:42.265014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.265015    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d85lz\" (UniqueName: \"kubernetes.io/projected/78877620-c79b-47d5-b64d-23ea310be7c2-kube-api-access-d85lz\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:42.526187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.526154    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:42.870590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.870539    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr" event={"ID":"cfd0dba9-afd5-42af-8aac-5c691917fd4f","Type":"ContainerStarted","Data":"6bfbfe221bf31d869f9b569d82a8da2cd0b8f647844bc5b0ae202cb850fbe6f1"}
Nov 03 13:38:42.871061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.870612    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr" event={"ID":"cfd0dba9-afd5-42af-8aac-5c691917fd4f","Type":"ContainerStarted","Data":"3ba4c0384d93ba379dca4cff840b919425de737ef96342e5a9d85a7e5020ab31"}
Nov 03 13:38:42.872186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.872158    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k" event={"ID":"78877620-c79b-47d5-b64d-23ea310be7c2","Type":"ContainerDied","Data":"2bd3c13814950d0b1673e51226c3d7c2dc5d1468439cc2c60e5fd900e6add44b"}
Nov 03 13:38:42.872186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.872175    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"
Nov 03 13:38:42.872405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.872201    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd3c13814950d0b1673e51226c3d7c2dc5d1468439cc2c60e5fd900e6add44b"
Nov 03 13:38:42.893463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:42.893423    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr" podStartSLOduration=1.893410316 podStartE2EDuration="1.893410316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:42.890206325 +0000 UTC m=+1763.967971361" watchObservedRunningTime="2026-11-03 13:38:42.893410316 +0000 UTC m=+1763.971175342"
Nov 03 13:38:43.219825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:43.219785    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:43.912546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:43.912511    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:43.918161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:43.918128    5420 generic.go:356] "Generic (PLEG): container finished" podID="cfd0dba9-afd5-42af-8aac-5c691917fd4f" containerID="6bfbfe221bf31d869f9b569d82a8da2cd0b8f647844bc5b0ae202cb850fbe6f1" exitCode=0
Nov 03 13:38:43.918323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:43.918167    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr" event={"ID":"cfd0dba9-afd5-42af-8aac-5c691917fd4f","Type":"ContainerDied","Data":"6bfbfe221bf31d869f9b569d82a8da2cd0b8f647844bc5b0ae202cb850fbe6f1"}
Nov 03 13:38:44.502015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:44.501981    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:44.953101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:44.953070    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:45.119520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.119498    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:45.146666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.146633    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:45.211497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.211461    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-openshift-ca\") pod \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") "
Nov 03 13:38:45.211688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.211568    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-xks-ca\") pod \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") "
Nov 03 13:38:45.211762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.211745    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd0dba9-afd5-42af-8aac-5c691917fd4f-kube-api-access-mbhbs\" (UniqueName: \"kubernetes.io/projected/cfd0dba9-afd5-42af-8aac-5c691917fd4f-kube-api-access-mbhbs\") pod \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\" (UID: \"cfd0dba9-afd5-42af-8aac-5c691917fd4f\") "
Nov 03 13:38:45.211834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.211807    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-openshift-ca" pod "cfd0dba9-afd5-42af-8aac-5c691917fd4f" (UID: "cfd0dba9-afd5-42af-8aac-5c691917fd4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:38:45.212380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.212357    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:45.213765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.213739    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-xks-ca" pod "cfd0dba9-afd5-42af-8aac-5c691917fd4f" (UID: "cfd0dba9-afd5-42af-8aac-5c691917fd4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:38:45.213863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.213841    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd0dba9-afd5-42af-8aac-5c691917fd4f-kube-api-access-mbhbs" pod "cfd0dba9-afd5-42af-8aac-5c691917fd4f" (UID: "cfd0dba9-afd5-42af-8aac-5c691917fd4f"). InnerVolumeSpecName "kube-api-access-mbhbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:38:45.313376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.313344    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbhbs\" (UniqueName: \"kubernetes.io/projected/cfd0dba9-afd5-42af-8aac-5c691917fd4f-kube-api-access-mbhbs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:45.313376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.313375    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0dba9-afd5-42af-8aac-5c691917fd4f-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:38:45.530498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.530391    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:45.936176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.936091    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"
Nov 03 13:38:45.936176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.936122    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr" event={"ID":"cfd0dba9-afd5-42af-8aac-5c691917fd4f","Type":"ContainerDied","Data":"3ba4c0384d93ba379dca4cff840b919425de737ef96342e5a9d85a7e5020ab31"}
Nov 03 13:38:45.936176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:45.936176    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ba4c0384d93ba379dca4cff840b919425de737ef96342e5a9d85a7e5020ab31"
Nov 03 13:38:46.235202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:46.235157    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:46.950893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:46.950861    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:47.851167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:47.851135    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:48.720790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:48.720566    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:48.922248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:48.922177    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:49.671915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:49.671882    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:50.053191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:50.053156    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:50.445138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:50.445045    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:51.463677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:51.463644    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:52.215877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:52.215843    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:52.904922 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:52.904888    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:53.493279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:53.489179    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:54.101972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:54.101934    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:54.721473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:54.721425    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:54.765180 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:38:54.765133    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:38:55.270258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:55.270194    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:55.416244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:55.415754    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:55.998897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:55.998862    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:56.591478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:56.591429    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:57.222143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:57.222108    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:57.836760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:57.836729    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:58.419573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:58.419533    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:59.074799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:59.074768    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:59.746324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:38:59.746273    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:00.409753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:00.409714    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:01.101804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:01.101769    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:01.730149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:01.730115    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:02.358280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:02.358245    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:02.847653 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:02.847613    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:39:02.941823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:02.941782    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:03.677873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:03.677844    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:04.473904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:04.473872    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:05.538018 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:05.537978    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:06.049575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:06.049540    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"]
Nov 03 13:39:06.084248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:06.083519    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba22c82-rzzvq"]
Nov 03 13:39:06.395814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:06.395726    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:06.575355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:06.575315    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:39:06.575850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:06.575390    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:39:06.612635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:06.612604    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:06.768123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:06.768086    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:07.086053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:07.085981    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69dbdf79-1c7d-43d8-a73f-24225cecb025" path="/var/lib/kubelet/pods/69dbdf79-1c7d-43d8-a73f-24225cecb025/volumes"
Nov 03 13:39:07.121298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:07.121262    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:07.931407 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:07.931379    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:08.623613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:08.623511    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:09.578061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.578028    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:09.725132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.725093    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:09.924196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.924120    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"]
Nov 03 13:39:09.924852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.924836    5420 cpu_manager.go:414] "containerMap: removing container" podUID="cfd0dba9-afd5-42af-8aac-5c691917fd4f" containerName="revoke-keys"
Nov 03 13:39:09.924902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.924854    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd0dba9-afd5-42af-8aac-5c691917fd4f" containerName="revoke-keys"
Nov 03 13:39:09.924902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.924881    5420 cpu_manager.go:414] "containerMap: removing container" podUID="78877620-c79b-47d5-b64d-23ea310be7c2" containerName="revoke-keys"
Nov 03 13:39:09.924902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.924886    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78877620-c79b-47d5-b64d-23ea310be7c2" containerName="revoke-keys"
Nov 03 13:39:09.925002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.924980    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd0dba9-afd5-42af-8aac-5c691917fd4f" containerName="revoke-keys"
Nov 03 13:39:09.925002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.924994    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="78877620-c79b-47d5-b64d-23ea310be7c2" containerName="revoke-keys"
Nov 03 13:39:09.928811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.928791    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:09.934595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.934572    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:39:09.934765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.934628    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:39:09.938400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.938375    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:39:09.938511 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.938416    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:39:09.940490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.940461    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"]
Nov 03 13:39:09.979253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.979206    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f98368b-49lh9\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:09.979425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.979325    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f98368b-49lh9\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:09.979501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:09.979485    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngb9\" (UniqueName: \"kubernetes.io/projected/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-kube-api-access-xngb9\") pod \"maas-api-revoke-sub-7f98368b-49lh9\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:10.080973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:10.080940    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f98368b-49lh9\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:10.081170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:10.081016    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f98368b-49lh9\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:10.081170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:10.081152    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngb9\" (UniqueName: \"kubernetes.io/projected/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-kube-api-access-xngb9\") pod \"maas-api-revoke-sub-7f98368b-49lh9\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:10.081593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:10.081572    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f98368b-49lh9\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:10.083426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:10.083405    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f98368b-49lh9\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:10.089859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:10.089840    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngb9\" (UniqueName: \"kubernetes.io/projected/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-kube-api-access-xngb9\") pod \"maas-api-revoke-sub-7f98368b-49lh9\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:10.247312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:10.247280    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:10.356596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:10.354426    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:10.561312 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:39:10.561279    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d3b619_2ccb_4e8c_9c56_d05553bd56df.slice/crio-3c7d9c14ea8c512ae1c39a648374dd0ec7620bde49952a043f70f7ea28dd6731 WatchSource:0}: Error finding container 3c7d9c14ea8c512ae1c39a648374dd0ec7620bde49952a043f70f7ea28dd6731: Status 404 returned error can't find the container with id 3c7d9c14ea8c512ae1c39a648374dd0ec7620bde49952a043f70f7ea28dd6731
Nov 03 13:39:10.561917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:10.561892    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"]
Nov 03 13:39:11.027942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:11.027901    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:11.244049 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:11.244018    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9" event={"ID":"c7d3b619-2ccb-4e8c-9c56-d05553bd56df","Type":"ContainerStarted","Data":"2b0f5c493a71f346be91da05005169b7a71b0b9e2913ca2fe1725ab9748f3ed1"}
Nov 03 13:39:11.244250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:11.244075    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9" event={"ID":"c7d3b619-2ccb-4e8c-9c56-d05553bd56df","Type":"ContainerStarted","Data":"3c7d9c14ea8c512ae1c39a648374dd0ec7620bde49952a043f70f7ea28dd6731"}
Nov 03 13:39:11.265615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:11.265563    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9" podStartSLOduration=2.265550036 podStartE2EDuration="2.265550036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:11.262837008 +0000 UTC m=+1792.340602034" watchObservedRunningTime="2026-11-03 13:39:11.265550036 +0000 UTC m=+1792.343315058"
Nov 03 13:39:11.587296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:11.587266    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:11.734976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:11.734945    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:12.254183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:12.254148    5420 generic.go:356] "Generic (PLEG): container finished" podID="c7d3b619-2ccb-4e8c-9c56-d05553bd56df" containerID="2b0f5c493a71f346be91da05005169b7a71b0b9e2913ca2fe1725ab9748f3ed1" exitCode=0
Nov 03 13:39:12.254649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:12.254229    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9" event={"ID":"c7d3b619-2ccb-4e8c-9c56-d05553bd56df","Type":"ContainerDied","Data":"2b0f5c493a71f346be91da05005169b7a71b0b9e2913ca2fe1725ab9748f3ed1"}
Nov 03 13:39:12.314319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:12.314284    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:12.934070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:12.933957    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:13.435357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.435331    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:13.546931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.546845    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-openshift-ca\") pod \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") "
Nov 03 13:39:13.547107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.546942    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-kube-api-access-xngb9\" (UniqueName: \"kubernetes.io/projected/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-kube-api-access-xngb9\") pod \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") "
Nov 03 13:39:13.547107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.547064    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-xks-ca\") pod \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\" (UID: \"c7d3b619-2ccb-4e8c-9c56-d05553bd56df\") "
Nov 03 13:39:13.547262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.547189    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-openshift-ca" pod "c7d3b619-2ccb-4e8c-9c56-d05553bd56df" (UID: "c7d3b619-2ccb-4e8c-9c56-d05553bd56df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:39:13.547624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.547604    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:13.548977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.548954    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-xks-ca" pod "c7d3b619-2ccb-4e8c-9c56-d05553bd56df" (UID: "c7d3b619-2ccb-4e8c-9c56-d05553bd56df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:39:13.549094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.549020    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-kube-api-access-xngb9" pod "c7d3b619-2ccb-4e8c-9c56-d05553bd56df" (UID: "c7d3b619-2ccb-4e8c-9c56-d05553bd56df"). InnerVolumeSpecName "kube-api-access-xngb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:13.597084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.596986    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:13.649901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.649856    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xngb9\" (UniqueName: \"kubernetes.io/projected/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-kube-api-access-xngb9\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:13.649901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:13.649896    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7d3b619-2ccb-4e8c-9c56-d05553bd56df-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:14.227232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:14.226796    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:14.273551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:14.273523    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"
Nov 03 13:39:14.273722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:14.273550    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9" event={"ID":"c7d3b619-2ccb-4e8c-9c56-d05553bd56df","Type":"ContainerDied","Data":"3c7d9c14ea8c512ae1c39a648374dd0ec7620bde49952a043f70f7ea28dd6731"}
Nov 03 13:39:14.273722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:14.273601    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c7d9c14ea8c512ae1c39a648374dd0ec7620bde49952a043f70f7ea28dd6731"
Nov 03 13:39:14.493328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:14.493251    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"]
Nov 03 13:39:14.507162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:14.507134    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"]
Nov 03 13:39:14.642741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:14.642710    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:15.120774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:15.120741    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:15.269551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:15.269521    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:15.729789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:15.729745    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:16.368621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:16.368586    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:17.151041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:17.150947    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:17.827909 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:17.827869    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:18.501404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:18.501314    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:19.107260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:19.103751    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:19.434948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:19.434844    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:19.547195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:19.547159    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:39:19.563905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:19.563885    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:39:19.753442 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:19.753398    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:20.194143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:20.194069    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:20.289400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:20.289370    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"]
Nov 03 13:39:20.352860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:20.352828    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:20.523152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:20.523121    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:20.748494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:20.747966    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:21.454911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:21.454879    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:22.118622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.118590    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:22.334765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.334736    5420 scope.go:122] "RemoveContainer" containerID="8f87f0c103224985819662702bdf8075bb15063bddc03bff8540523d7c70cc44"
Nov 03 13:39:22.541665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.541633    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"]
Nov 03 13:39:22.542354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.542340    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c7d3b619-2ccb-4e8c-9c56-d05553bd56df" containerName="revoke-keys"
Nov 03 13:39:22.542416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.542355    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d3b619-2ccb-4e8c-9c56-d05553bd56df" containerName="revoke-keys"
Nov 03 13:39:22.542479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.542470    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d3b619-2ccb-4e8c-9c56-d05553bd56df" containerName="revoke-keys"
Nov 03 13:39:22.547850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.547822    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.553139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.553115    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:39:22.553274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.553168    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:39:22.554054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.554035    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"]
Nov 03 13:39:22.579405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.579380    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:39:22.579563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.579404    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:39:22.700735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.700631    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd21913-xkds4\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.702682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.702649    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfzd\" (UniqueName: \"kubernetes.io/projected/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-kube-api-access-2kfzd\") pod \"maas-api-revoke-sub-edd21913-xkds4\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.702682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.702662    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:22.702980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.702959    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd21913-xkds4\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.804670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.804589    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd21913-xkds4\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.804826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.804704    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfzd\" (UniqueName: \"kubernetes.io/projected/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-kube-api-access-2kfzd\") pod \"maas-api-revoke-sub-edd21913-xkds4\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.804826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.804726    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd21913-xkds4\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.805280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.805261    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd21913-xkds4\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.807099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.807077    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd21913-xkds4\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.813947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.813926    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfzd\" (UniqueName: \"kubernetes.io/projected/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-kube-api-access-2kfzd\") pod \"maas-api-revoke-sub-edd21913-xkds4\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:22.839654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.839630    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:22.864551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:22.864531    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:23.155982 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:39:23.155955    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf4ddf1_b7c5_4fc1_8bee_be892405db47.slice/crio-fd69057cadc133ed6fa240c6bd94eacae640e558a038257e4dfc116554518aa8 WatchSource:0}: Error finding container fd69057cadc133ed6fa240c6bd94eacae640e558a038257e4dfc116554518aa8: Status 404 returned error can't find the container with id fd69057cadc133ed6fa240c6bd94eacae640e558a038257e4dfc116554518aa8
Nov 03 13:39:23.158631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:23.158603    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"]
Nov 03 13:39:23.374015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:23.373349    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:23.380705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:23.380674    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4" event={"ID":"7cf4ddf1-b7c5-4fc1-8bee-be892405db47","Type":"ContainerStarted","Data":"fd69057cadc133ed6fa240c6bd94eacae640e558a038257e4dfc116554518aa8"}
Nov 03 13:39:23.930547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:23.926834    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:24.390273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:24.390240    5420 generic.go:356] "Generic (PLEG): container finished" podID="7cf4ddf1-b7c5-4fc1-8bee-be892405db47" containerID="d29e279166ec2b2f66fb2492c6c9258691dc36fc39cacd53cec309eb576b121e" exitCode=0
Nov 03 13:39:24.390721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:24.390372    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4" event={"ID":"7cf4ddf1-b7c5-4fc1-8bee-be892405db47","Type":"ContainerDied","Data":"d29e279166ec2b2f66fb2492c6c9258691dc36fc39cacd53cec309eb576b121e"}
Nov 03 13:39:24.608279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:24.607148    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:24.732324 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:39:24.732286    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:39:25.289820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.289774    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts" podUID="89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" containerName="payload-pre-processing" containerID="cri-o://c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7" gracePeriod=25
Nov 03 13:39:25.526205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.526173    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:25.599306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.599276    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:25.646173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.646132    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-xks-ca\") pod \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") "
Nov 03 13:39:25.646378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.646315    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-kube-api-access-2kfzd\" (UniqueName: \"kubernetes.io/projected/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-kube-api-access-2kfzd\") pod \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") "
Nov 03 13:39:25.646463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.646375    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-openshift-ca\") pod \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\" (UID: \"7cf4ddf1-b7c5-4fc1-8bee-be892405db47\") "
Nov 03 13:39:25.646837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.646803    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-openshift-ca" pod "7cf4ddf1-b7c5-4fc1-8bee-be892405db47" (UID: "7cf4ddf1-b7c5-4fc1-8bee-be892405db47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:39:25.647334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.647309    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:25.648478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.648448    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-xks-ca" pod "7cf4ddf1-b7c5-4fc1-8bee-be892405db47" (UID: "7cf4ddf1-b7c5-4fc1-8bee-be892405db47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:39:25.648998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.648979    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-kube-api-access-2kfzd" pod "7cf4ddf1-b7c5-4fc1-8bee-be892405db47" (UID: "7cf4ddf1-b7c5-4fc1-8bee-be892405db47"). InnerVolumeSpecName "kube-api-access-2kfzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:25.748296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.748265    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kfzd\" (UniqueName: \"kubernetes.io/projected/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-kube-api-access-2kfzd\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:25.748296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:25.748293    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4ddf1-b7c5-4fc1-8bee-be892405db47-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:26.187424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:26.187387    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:26.264814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:26.264782    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"]
Nov 03 13:39:26.291328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:26.291301    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"]
Nov 03 13:39:26.408264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:26.408164    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"
Nov 03 13:39:26.408264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:26.408177    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4" event={"ID":"7cf4ddf1-b7c5-4fc1-8bee-be892405db47","Type":"ContainerDied","Data":"fd69057cadc133ed6fa240c6bd94eacae640e558a038257e4dfc116554518aa8"}
Nov 03 13:39:26.408264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:26.408255    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd69057cadc133ed6fa240c6bd94eacae640e558a038257e4dfc116554518aa8"
Nov 03 13:39:26.434641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:26.434614    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:26.649293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:26.649232    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:27.107974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:27.107944    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:27.633287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:27.633253    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:27.687501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:27.687469    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts" podUID="89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:39:28.021675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:28.021645    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:28.327745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:28.327674    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:28.720033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:28.719962    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:29.292814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.292786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:29.438683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.438642    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:29.515422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.515368    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main" containerID="cri-o://252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da" gracePeriod=45
Nov 03 13:39:29.531272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.531205    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main" containerID="cri-o://19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0" gracePeriod=45
Nov 03 13:39:29.823249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.823056    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:39:29.868132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.868105    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:39:29.951667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.951639    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-kube-api-access-xr8bb\" (UniqueName: \"kubernetes.io/projected/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-kube-api-access-xr8bb\") pod \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") "
Nov 03 13:39:29.951863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.951723    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-home\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-home\") pod \"68c430da-26cd-420a-85de-84e4187fe5d0\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") "
Nov 03 13:39:29.951863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.951746    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tls-certs\") pod \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") "
Nov 03 13:39:29.951863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.951814    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-home\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-home\") pod \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") "
Nov 03 13:39:29.952058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.951870    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-tmp-dir\") pod \"68c430da-26cd-420a-85de-84e4187fe5d0\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") "
Nov 03 13:39:29.952058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.951910    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-model-cache\") pod \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") "
Nov 03 13:39:29.952058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.951938    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-dshm\") pod \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") "
Nov 03 13:39:29.952058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.951990    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tmp-dir\") pod \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\" (UID: \"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a\") "
Nov 03 13:39:29.952058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952055    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c430da-26cd-420a-85de-84e4187fe5d0-kube-api-access-t6cgs\" (UniqueName: \"kubernetes.io/projected/68c430da-26cd-420a-85de-84e4187fe5d0-kube-api-access-t6cgs\") pod \"68c430da-26cd-420a-85de-84e4187fe5d0\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") "
Nov 03 13:39:29.952384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952066    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-home" pod "68c430da-26cd-420a-85de-84e4187fe5d0" (UID: "68c430da-26cd-420a-85de-84e4187fe5d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.952384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952158    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-tmp-dir" pod "68c430da-26cd-420a-85de-84e4187fe5d0" (UID: "68c430da-26cd-420a-85de-84e4187fe5d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.952384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952170    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68c430da-26cd-420a-85de-84e4187fe5d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/68c430da-26cd-420a-85de-84e4187fe5d0-tls-certs\") pod \"68c430da-26cd-420a-85de-84e4187fe5d0\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") "
Nov 03 13:39:29.952384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952338    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-model-cache\") pod \"68c430da-26cd-420a-85de-84e4187fe5d0\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") "
Nov 03 13:39:29.952384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952378    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-dshm\") pod \"68c430da-26cd-420a-85de-84e4187fe5d0\" (UID: \"68c430da-26cd-420a-85de-84e4187fe5d0\") "
Nov 03 13:39:29.952670 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952506    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-home" pod "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" (UID: "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.952730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952675    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tmp-dir" pod "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" (UID: "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.952730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952695    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-model-cache" pod "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" (UID: "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.952881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.952862    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-model-cache" pod "68c430da-26cd-420a-85de-84e4187fe5d0" (UID: "68c430da-26cd-420a-85de-84e4187fe5d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.953236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.953196    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.953351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.953242    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.953351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.953256    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.953351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.953272    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.953351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.953287    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.953351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.953300    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:29.954721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.954692    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c430da-26cd-420a-85de-84e4187fe5d0-kube-api-access-t6cgs" pod "68c430da-26cd-420a-85de-84e4187fe5d0" (UID: "68c430da-26cd-420a-85de-84e4187fe5d0"). InnerVolumeSpecName "kube-api-access-t6cgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:29.954721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.954699    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c430da-26cd-420a-85de-84e4187fe5d0-tls-certs" pod "68c430da-26cd-420a-85de-84e4187fe5d0" (UID: "68c430da-26cd-420a-85de-84e4187fe5d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:39:29.954901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.954747    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-kube-api-access-xr8bb" pod "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" (UID: "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a"). InnerVolumeSpecName "kube-api-access-xr8bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:29.954901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.954872    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-dshm" pod "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" (UID: "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.955039 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.954948    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-dshm" pod "68c430da-26cd-420a-85de-84e4187fe5d0" (UID: "68c430da-26cd-420a-85de-84e4187fe5d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:29.955501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:29.955477    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tls-certs" pod "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" (UID: "ca6dd9ae-3066-4739-9a4c-fe8b3de3493a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:39:30.041586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.040104    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:30.057683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.057543    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c430da-26cd-420a-85de-84e4187fe5d0-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:30.057683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.057582    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8bb\" (UniqueName: \"kubernetes.io/projected/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-kube-api-access-xr8bb\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:30.057683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.057598    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:30.057683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.057613    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:30.057683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.057628    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6cgs\" (UniqueName: \"kubernetes.io/projected/68c430da-26cd-420a-85de-84e4187fe5d0-kube-api-access-t6cgs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:30.057683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.057642    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c430da-26cd-420a-85de-84e4187fe5d0-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:30.448844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.448806    5420 generic.go:356] "Generic (PLEG): container finished" podID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerID="19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0" exitCode=0
Nov 03 13:39:30.449027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.448855    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"
Nov 03 13:39:30.449027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.448889    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" event={"ID":"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a","Type":"ContainerDied","Data":"19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0"}
Nov 03 13:39:30.449027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.448935    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv" event={"ID":"ca6dd9ae-3066-4739-9a4c-fe8b3de3493a","Type":"ContainerDied","Data":"9d235635c287edc395f397a3f4e9d511cedf090ebabcfbe71d4a40369ffad4bb"}
Nov 03 13:39:30.449027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.448958    5420 scope.go:122] "RemoveContainer" containerID="19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0"
Nov 03 13:39:30.450490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.450468    5420 generic.go:356] "Generic (PLEG): container finished" podID="68c430da-26cd-420a-85de-84e4187fe5d0" containerID="252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da" exitCode=0
Nov 03 13:39:30.450611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.450508    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"
Nov 03 13:39:30.450611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.450545    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" event={"ID":"68c430da-26cd-420a-85de-84e4187fe5d0","Type":"ContainerDied","Data":"252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da"}
Nov 03 13:39:30.450611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.450593    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp" event={"ID":"68c430da-26cd-420a-85de-84e4187fe5d0","Type":"ContainerDied","Data":"cbf721fddcd776b2edebcb8a693302ab44124580fde37eee21f8475e8419992b"}
Nov 03 13:39:30.461122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.461106    5420 scope.go:122] "RemoveContainer" containerID="19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0"
Nov 03 13:39:30.461439 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:39:30.461417    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0\": container with ID starting with 19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0 not found: ID does not exist" containerID="19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0"
Nov 03 13:39:30.461505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.461451    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0"} err="failed to get container status \"19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0\": rpc error: code = NotFound desc = could not find container \"19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0\": container with ID starting with 19b5102060cb1d4e74fa4b775be699292b29cd21827427582fb0e2bd6ec274c0 not found: ID does not exist"
Nov 03 13:39:30.461505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.461475    5420 scope.go:122] "RemoveContainer" containerID="252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da"
Nov 03 13:39:30.471350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.471317    5420 scope.go:122] "RemoveContainer" containerID="252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da"
Nov 03 13:39:30.471579 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:39:30.471556    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da\": container with ID starting with 252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da not found: ID does not exist" containerID="252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da"
Nov 03 13:39:30.471661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.471588    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da"} err="failed to get container status \"252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da\": rpc error: code = NotFound desc = could not find container \"252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da\": container with ID starting with 252a697723370e145db66053445b721598d22e781bd753be01a7402a284a45da not found: ID does not exist"
Nov 03 13:39:30.485323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.485299    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"]
Nov 03 13:39:30.492536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.492516    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/simulator-w4-c381a7-kserve-765c79788-fgssv"]
Nov 03 13:39:30.505188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.505165    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"]
Nov 03 13:39:30.509562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.509538    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c381a7/premium-w4-c381a7-kserve-66b85f8b97-cdhpp"]
Nov 03 13:39:30.726263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:30.725840    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:31.086159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.086086    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" path="/var/lib/kubelet/pods/68c430da-26cd-420a-85de-84e4187fe5d0/volumes"
Nov 03 13:39:31.086627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.086610    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" path="/var/lib/kubelet/pods/ca6dd9ae-3066-4739-9a4c-fe8b3de3493a/volumes"
Nov 03 13:39:31.265769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.265734    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4" podUID="30568271-6049-4af4-bf3a-39659d975931" containerName="payload-processing" containerID="cri-o://0fe3953c5f6f965362c9d9df365f7c3c673f9a4b20e04f10c874fb609f423a02" gracePeriod=25
Nov 03 13:39:31.292354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.292324    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g" podUID="539260d3-0a89-434f-a1a5-44890179b602" containerName="payload-pre-processing" containerID="cri-o://f8250ef7498a596657730522b435a98cd102ca5169f699d83a6a1edb0e51e7e8" gracePeriod=25
Nov 03 13:39:31.451342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.451310    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:31.475192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.474944    5420 generic.go:356] "Generic (PLEG): container finished" podID="539260d3-0a89-434f-a1a5-44890179b602" containerID="f8250ef7498a596657730522b435a98cd102ca5169f699d83a6a1edb0e51e7e8" exitCode=0
Nov 03 13:39:31.475192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.475109    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g" event={"ID":"539260d3-0a89-434f-a1a5-44890179b602","Type":"ContainerDied","Data":"f8250ef7498a596657730522b435a98cd102ca5169f699d83a6a1edb0e51e7e8"}
Nov 03 13:39:31.482662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.482526    5420 generic.go:356] "Generic (PLEG): container finished" podID="30568271-6049-4af4-bf3a-39659d975931" containerID="0fe3953c5f6f965362c9d9df365f7c3c673f9a4b20e04f10c874fb609f423a02" exitCode=0
Nov 03 13:39:31.482982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.482916    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4" event={"ID":"30568271-6049-4af4-bf3a-39659d975931","Type":"ContainerDied","Data":"0fe3953c5f6f965362c9d9df365f7c3c673f9a4b20e04f10c874fb609f423a02"}
Nov 03 13:39:31.632964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.632940    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:39:31.704428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.704398    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:39:31.783294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.783266    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539260d3-0a89-434f-a1a5-44890179b602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539260d3-0a89-434f-a1a5-44890179b602-plugins-config-volume\") pod \"539260d3-0a89-434f-a1a5-44890179b602\" (UID: \"539260d3-0a89-434f-a1a5-44890179b602\") "
Nov 03 13:39:31.783462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.783337    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539260d3-0a89-434f-a1a5-44890179b602-kube-api-access-ckdzb\" (UniqueName: \"kubernetes.io/projected/539260d3-0a89-434f-a1a5-44890179b602-kube-api-access-ckdzb\") pod \"539260d3-0a89-434f-a1a5-44890179b602\" (UID: \"539260d3-0a89-434f-a1a5-44890179b602\") "
Nov 03 13:39:31.783462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.783395    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30568271-6049-4af4-bf3a-39659d975931-kube-api-access-b55x2\" (UniqueName: \"kubernetes.io/projected/30568271-6049-4af4-bf3a-39659d975931-kube-api-access-b55x2\") pod \"30568271-6049-4af4-bf3a-39659d975931\" (UID: \"30568271-6049-4af4-bf3a-39659d975931\") "
Nov 03 13:39:31.783547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.783519    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30568271-6049-4af4-bf3a-39659d975931-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30568271-6049-4af4-bf3a-39659d975931-plugins-config-volume\") pod \"30568271-6049-4af4-bf3a-39659d975931\" (UID: \"30568271-6049-4af4-bf3a-39659d975931\") "
Nov 03 13:39:31.783841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.783807    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539260d3-0a89-434f-a1a5-44890179b602-plugins-config-volume" pod "539260d3-0a89-434f-a1a5-44890179b602" (UID: "539260d3-0a89-434f-a1a5-44890179b602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:39:31.783989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.783868    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30568271-6049-4af4-bf3a-39659d975931-plugins-config-volume" pod "30568271-6049-4af4-bf3a-39659d975931" (UID: "30568271-6049-4af4-bf3a-39659d975931"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:39:31.784300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.784279    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30568271-6049-4af4-bf3a-39659d975931-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:31.784396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.784304    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539260d3-0a89-434f-a1a5-44890179b602-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:31.785791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.785766    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539260d3-0a89-434f-a1a5-44890179b602-kube-api-access-ckdzb" pod "539260d3-0a89-434f-a1a5-44890179b602" (UID: "539260d3-0a89-434f-a1a5-44890179b602"). InnerVolumeSpecName "kube-api-access-ckdzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:31.786094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.786073    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30568271-6049-4af4-bf3a-39659d975931-kube-api-access-b55x2" pod "30568271-6049-4af4-bf3a-39659d975931" (UID: "30568271-6049-4af4-bf3a-39659d975931"). InnerVolumeSpecName "kube-api-access-b55x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:31.886130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.886059    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckdzb\" (UniqueName: \"kubernetes.io/projected/539260d3-0a89-434f-a1a5-44890179b602-kube-api-access-ckdzb\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:31.886130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:31.886088    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b55x2\" (UniqueName: \"kubernetes.io/projected/30568271-6049-4af4-bf3a-39659d975931-kube-api-access-b55x2\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:32.245222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.245166    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:32.495357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.495274    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"
Nov 03 13:39:32.495357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.495315    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g" event={"ID":"539260d3-0a89-434f-a1a5-44890179b602","Type":"ContainerDied","Data":"60df2f3c9678ae7ee938b7433181ce760fa69d652c58112a0b9237c5e0a99a5c"}
Nov 03 13:39:32.495599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.495376    5420 scope.go:122] "RemoveContainer" containerID="f8250ef7498a596657730522b435a98cd102ca5169f699d83a6a1edb0e51e7e8"
Nov 03 13:39:32.496946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.496924    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"
Nov 03 13:39:32.497088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.496940    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4" event={"ID":"30568271-6049-4af4-bf3a-39659d975931","Type":"ContainerDied","Data":"a784ade5bb34f061fb0781a6b248b5de103a99c1e5042051c581e75be8c96323"}
Nov 03 13:39:32.519584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.519562    5420 scope.go:122] "RemoveContainer" containerID="0fe3953c5f6f965362c9d9df365f7c3c673f9a4b20e04f10c874fb609f423a02"
Nov 03 13:39:32.536842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.536817    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"]
Nov 03 13:39:32.541155 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.541134    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6f33a0-7fc4449fc5-rg4b4"]
Nov 03 13:39:32.556193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.556163    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"]
Nov 03 13:39:32.560106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:32.560086    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6f33a0-7dddf5fbf9-5r75g"]
Nov 03 13:39:33.031312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.031279    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:33.085490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.085460    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30568271-6049-4af4-bf3a-39659d975931" path="/var/lib/kubelet/pods/30568271-6049-4af4-bf3a-39659d975931/volumes"
Nov 03 13:39:33.085953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.085937    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539260d3-0a89-434f-a1a5-44890179b602" path="/var/lib/kubelet/pods/539260d3-0a89-434f-a1a5-44890179b602/volumes"
Nov 03 13:39:33.425044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.424961    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"]
Nov 03 13:39:33.426050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426018    5420 cpu_manager.go:414] "containerMap: removing container" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main"
Nov 03 13:39:33.426050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426047    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main"
Nov 03 13:39:33.426296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426078    5420 cpu_manager.go:414] "containerMap: removing container" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main"
Nov 03 13:39:33.426296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426086    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main"
Nov 03 13:39:33.426296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426104    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7cf4ddf1-b7c5-4fc1-8bee-be892405db47" containerName="revoke-keys"
Nov 03 13:39:33.426296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426114    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf4ddf1-b7c5-4fc1-8bee-be892405db47" containerName="revoke-keys"
Nov 03 13:39:33.426296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426131    5420 cpu_manager.go:414] "containerMap: removing container" podUID="30568271-6049-4af4-bf3a-39659d975931" containerName="payload-processing"
Nov 03 13:39:33.426296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426138    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30568271-6049-4af4-bf3a-39659d975931" containerName="payload-processing"
Nov 03 13:39:33.426296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426169    5420 cpu_manager.go:414] "containerMap: removing container" podUID="539260d3-0a89-434f-a1a5-44890179b602" containerName="payload-pre-processing"
Nov 03 13:39:33.426296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426177    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539260d3-0a89-434f-a1a5-44890179b602" containerName="payload-pre-processing"
Nov 03 13:39:33.426742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426368    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6dd9ae-3066-4739-9a4c-fe8b3de3493a" containerName="main"
Nov 03 13:39:33.426742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426386    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c430da-26cd-420a-85de-84e4187fe5d0" containerName="main"
Nov 03 13:39:33.426742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426396    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf4ddf1-b7c5-4fc1-8bee-be892405db47" containerName="revoke-keys"
Nov 03 13:39:33.426742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426405    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="30568271-6049-4af4-bf3a-39659d975931" containerName="payload-processing"
Nov 03 13:39:33.426742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.426419    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="539260d3-0a89-434f-a1a5-44890179b602" containerName="payload-pre-processing"
Nov 03 13:39:33.431837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.431817    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.432781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.432753    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.432896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.432874    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.432955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.432928    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.433053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.433034    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.433147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.433130    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qdlg\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-kube-api-access-2qdlg\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.433244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.433160    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.433244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.433195    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.433367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.433324    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.433424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.433369    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4734f2-cf4e-41d1-a84f-8a606818d391-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.480968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.450941    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"]
Nov 03 13:39:33.534467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534429    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.534649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534486    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qdlg\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-kube-api-access-2qdlg\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.534649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534506    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.534649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534533    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.534903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534662    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.534903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534702    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4734f2-cf4e-41d1-a84f-8a606818d391-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.534903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534845    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.534903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534889    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.535165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534907    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.535165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534947    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.535165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.534949    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.535165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.535018    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.535462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.535206    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.535529 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.535470    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4734f2-cf4e-41d1-a84f-8a606818d391-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.537290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.537265    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.537433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.537420    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.547083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.547050    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.547909 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.547887    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qdlg\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-kube-api-access-2qdlg\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.797152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.797109    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:33.996237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:33.987696    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:34.095676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:34.095651    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"]
Nov 03 13:39:34.096508 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:39:34.096476    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4734f2_cf4e_41d1_a84f_8a606818d391.slice/crio-0f5e56cbc33a19b4e46cb36006aeb4348a54bbdec0f9bfb5b4c623cbfd09d969 WatchSource:0}: Error finding container 0f5e56cbc33a19b4e46cb36006aeb4348a54bbdec0f9bfb5b4c623cbfd09d969: Status 404 returned error can't find the container with id 0f5e56cbc33a19b4e46cb36006aeb4348a54bbdec0f9bfb5b4c623cbfd09d969
Nov 03 13:39:34.099342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:34.099303    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:39:34.099428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:34.099391    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:39:34.099468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:34.099426    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:39:34.520181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:34.520145    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" event={"ID":"cc4734f2-cf4e-41d1-a84f-8a606818d391","Type":"ContainerStarted","Data":"be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862"}
Nov 03 13:39:34.520755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:34.520199    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" event={"ID":"cc4734f2-cf4e-41d1-a84f-8a606818d391","Type":"ContainerStarted","Data":"0f5e56cbc33a19b4e46cb36006aeb4348a54bbdec0f9bfb5b4c623cbfd09d969"}
Nov 03 13:39:34.561926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:34.561866    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" podStartSLOduration=1.561847494 podStartE2EDuration="1.561847494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:34.556104391 +0000 UTC m=+1815.633869419" watchObservedRunningTime="2026-11-03 13:39:34.561847494 +0000 UTC m=+1815.639612521"
Nov 03 13:39:34.798105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:34.798019    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:34.894748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:34.894715    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:35.788483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:35.788387    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:35.799884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:35.799653    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 13:39:35.799884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:35.799708    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 13:39:36.532808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:36.532775    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:36.797514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:36.797426    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 13:39:36.797514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:36.797477    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 13:39:36.844483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:36.844448    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:37.320738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:37.320707    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:37.635289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:37.634735    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:37.716855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:37.716435    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts" podUID="89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:39:37.798527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:37.798062    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 13:39:37.798527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:37.798123    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 13:39:37.826770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:37.826736    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:38.073242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.071837    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:38.300484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.300451    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:38.662519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.662485    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"]
Nov 03 13:39:38.667191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.667169    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.675957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.675932    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-cd80b3-openshift-default-dockercfg-h6j9x\""
Nov 03 13:39:38.682504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.682471    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"]
Nov 03 13:39:38.777657 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.777625    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:38.819234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.816005    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7dqk\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-kube-api-access-h7dqk\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.830684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.830657    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-token\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.830993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.830973    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-socket\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.831245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.831200    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-envoy\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.831458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.831442    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-certs\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.831652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.831630    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ca244-2b0b-4de3-8b55-2834da4e839a-istiod-ca-cert\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.831808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.831793    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-credential-socket\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.831972 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.831957    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-podinfo\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.836232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.832364    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-data\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.836555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.830557    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 13:39:38.836715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.836691    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 13:39:38.946468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.946376    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-data\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.947225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.947182    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7dqk\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-kube-api-access-h7dqk\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.947435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.947417    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-token\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.947583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.947568    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-socket\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.947761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.947741    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-envoy\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.947918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.947902    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-certs\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.948082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.948064    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ca244-2b0b-4de3-8b55-2834da4e839a-istiod-ca-cert\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.950309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.950287    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-credential-socket\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.966171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.966141    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-podinfo\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.972110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.949573    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-socket\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.980234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.950187    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ca244-2b0b-4de3-8b55-2834da4e839a-istiod-ca-cert\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.992238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.947086    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-data\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.992238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.950707    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-credential-socket\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.992238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.949339    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-certs\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.992238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.981031    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7dqk\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-kube-api-access-h7dqk\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.992238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.981291    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-envoy\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.992238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.984241    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-podinfo\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:38.996238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:38.994956    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-token\") pod \"e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") " pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:39.201102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.200336    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:39.284812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.284782    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:39.349041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.348977    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:39.606783 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:39:39.606751    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186ca244_2b0b_4de3_8b55_2834da4e839a.slice/crio-89b25a5518a72de946245de0b00a9b37b2a92af453680b4e98ecfe86c567b099 WatchSource:0}: Error finding container 89b25a5518a72de946245de0b00a9b37b2a92af453680b4e98ecfe86c567b099: Status 404 returned error can't find the container with id 89b25a5518a72de946245de0b00a9b37b2a92af453680b4e98ecfe86c567b099
Nov 03 13:39:39.607083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.607062    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"]
Nov 03 13:39:39.609596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.609566    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:39:39.609662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.609629    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:39:39.609662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.609658    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:39:39.658739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.658706    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc" event={"ID":"186ca244-2b0b-4de3-8b55-2834da4e839a","Type":"ContainerStarted","Data":"89b25a5518a72de946245de0b00a9b37b2a92af453680b4e98ecfe86c567b099"}
Nov 03 13:39:39.797954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.797908    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 13:39:39.798110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:39.797982    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 13:39:40.067804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:40.067768    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:40.670655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:40.670614    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc" event={"ID":"186ca244-2b0b-4de3-8b55-2834da4e839a","Type":"ContainerStarted","Data":"6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4"}
Nov 03 13:39:40.710879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:40.710838    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:40.714178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:40.714119    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc" podStartSLOduration=2.71409693 podStartE2EDuration="2.71409693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:40.707030713 +0000 UTC m=+1821.784795740" watchObservedRunningTime="2026-11-03 13:39:40.71409693 +0000 UTC m=+1821.791861999"
Nov 03 13:39:40.802951 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:40.802917    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:40.803767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:40.803741    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:40.804446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:40.804424    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:39:41.166243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:41.166180    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:41.285168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:41.285141    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:41.291170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:41.291144    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:41.328328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:41.328299    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:41.504138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:41.504103    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:41.678825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:41.678790    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:41.679657 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:41.679634    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:42.422106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:42.422066    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:43.206238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:43.193369    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:43.435179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:43.435148    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:44.148878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:44.148777    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:44.823110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:44.819858    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:45.536956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.536880    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:45.617642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.617621    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:39:45.695441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.695413    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-plugins-config-volume\") pod \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\" (UID: \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\") "
Nov 03 13:39:45.695718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.695696    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-kube-api-access-2drw4\" (UniqueName: \"kubernetes.io/projected/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-kube-api-access-2drw4\") pod \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\" (UID: \"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65\") "
Nov 03 13:39:45.695831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.695811    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-plugins-config-volume" pod "89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" (UID: "89bd0a5a-58c6-4bd6-b731-c2c7743c4b65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:39:45.696307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.696290    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:45.697705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.697680    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-kube-api-access-2drw4" pod "89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" (UID: "89bd0a5a-58c6-4bd6-b731-c2c7743c4b65"). InnerVolumeSpecName "kube-api-access-2drw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:45.717838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.717802    5420 generic.go:356] "Generic (PLEG): container finished" podID="89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" containerID="c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7" exitCode=0
Nov 03 13:39:45.717941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.717840    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts" event={"ID":"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65","Type":"ContainerDied","Data":"c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7"}
Nov 03 13:39:45.717941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.717896    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"
Nov 03 13:39:45.717941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.717904    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts" event={"ID":"89bd0a5a-58c6-4bd6-b731-c2c7743c4b65","Type":"ContainerDied","Data":"528c934a2afc95afcb9076aa154833113ed8b4c2d46f3085a9a99b2cd7366580"}
Nov 03 13:39:45.717941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.717933    5420 scope.go:122] "RemoveContainer" containerID="c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7"
Nov 03 13:39:45.732302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.732278    5420 scope.go:122] "RemoveContainer" containerID="c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7"
Nov 03 13:39:45.732638 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:39:45.732612    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7\": container with ID starting with c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7 not found: ID does not exist" containerID="c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7"
Nov 03 13:39:45.732742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.732647    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7"} err="failed to get container status \"c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7\": rpc error: code = NotFound desc = could not find container \"c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7\": container with ID starting with c826260532456cc897d9318b7be7437df466dde2ac54eedde760979175b71de7 not found: ID does not exist"
Nov 03 13:39:45.759746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.759720    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"]
Nov 03 13:39:45.773363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.773337    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c381a7-6c77cbc9b7-fdfts"]
Nov 03 13:39:45.797302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.797278    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2drw4\" (UniqueName: \"kubernetes.io/projected/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65-kube-api-access-2drw4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:45.845033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:45.844941    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:46.578777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:46.578747    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:47.094085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:47.094047    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" path="/var/lib/kubelet/pods/89bd0a5a-58c6-4bd6-b731-c2c7743c4b65/volumes"
Nov 03 13:39:47.467347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:47.467302    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:47.824282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:47.808493    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:48.662166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:48.662129    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:49.417147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:49.417114    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:50.170533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:50.170501    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:50.840865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:50.840834    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:51.645651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:51.645611    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:51.847041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:51.847006    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:52.014784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:52.014753    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:52.316807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:52.310707    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:52.350861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:52.350804    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"]
Nov 03 13:39:52.351690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:52.351525    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc" podUID="186ca244-2b0b-4de3-8b55-2834da4e839a" containerName="istio-proxy" containerID="cri-o://6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4" gracePeriod=30
Nov 03 13:39:52.548345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:52.548314    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:52.737386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:52.737350    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:52.896870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:52.896842    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:53.462110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:53.462082    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:53.631901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:53.631870    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:54.387617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:54.387578    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:54.731429 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:39:54.731395    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:39:54.995450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:54.995344    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:55.190645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:55.180993    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:55.448454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:55.448368    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:55.717523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:55.717490    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:56.051185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.051105    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"]
Nov 03 13:39:56.051375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.051345    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy" containerID="cri-o://8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2" gracePeriod=30
Nov 03 13:39:56.055602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.055575    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"]
Nov 03 13:39:56.074031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.074008    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29ce8b3-lmtt7"]
Nov 03 13:39:56.100712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.100686    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"]
Nov 03 13:39:56.104928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.104903    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"]
Nov 03 13:39:56.110481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.110461    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ba37af-q86v2"]
Nov 03 13:39:56.114149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.114130    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3412a92-5rn7x"]
Nov 03 13:39:56.115358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.115313    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:56.290245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.289052    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:56.681408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.681369    5420 patch_prober.go:29] interesting pod/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 13:39:56.681794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:56.681442    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc" podUID="186ca244-2b0b-4de3-8b55-2834da4e839a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:39:57.009446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.009372    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:57.085523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.085494    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc96b77-2e71-4e30-8ef3-9bec7976814a" path="/var/lib/kubelet/pods/6fc96b77-2e71-4e30-8ef3-9bec7976814a/volumes"
Nov 03 13:39:57.085965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.085952    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1e7472-ef8a-43b3-b515-778af59b79f4" path="/var/lib/kubelet/pods/8b1e7472-ef8a-43b3-b515-778af59b79f4/volumes"
Nov 03 13:39:57.086387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.086374    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b113f1ba-d494-4300-a9ee-f979695a60c5" path="/var/lib/kubelet/pods/b113f1ba-d494-4300-a9ee-f979695a60c5/volumes"
Nov 03 13:39:57.653264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.653194    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:57.778604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.777602    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186ca244-2b0b-4de3-8b55-2834da4e839a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ca244-2b0b-4de3-8b55-2834da4e839a-istiod-ca-cert\") pod \"186ca244-2b0b-4de3-8b55-2834da4e839a\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") "
Nov 03 13:39:57.778604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.777776    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-data\") pod \"186ca244-2b0b-4de3-8b55-2834da4e839a\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") "
Nov 03 13:39:57.778604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.777812    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-socket\") pod \"186ca244-2b0b-4de3-8b55-2834da4e839a\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") "
Nov 03 13:39:57.778604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.777850    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-kube-api-access-h7dqk\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-kube-api-access-h7dqk\") pod \"186ca244-2b0b-4de3-8b55-2834da4e839a\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") "
Nov 03 13:39:57.778604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.777882    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-credential-socket\") pod \"186ca244-2b0b-4de3-8b55-2834da4e839a\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") "
Nov 03 13:39:57.778604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.777939    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-token\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-token\") pod \"186ca244-2b0b-4de3-8b55-2834da4e839a\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") "
Nov 03 13:39:57.778604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.777971    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-envoy\") pod \"186ca244-2b0b-4de3-8b55-2834da4e839a\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") "
Nov 03 13:39:57.778604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.778068    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-certs\") pod \"186ca244-2b0b-4de3-8b55-2834da4e839a\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") "
Nov 03 13:39:57.778604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.778152    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-podinfo\") pod \"186ca244-2b0b-4de3-8b55-2834da4e839a\" (UID: \"186ca244-2b0b-4de3-8b55-2834da4e839a\") "
Nov 03 13:39:57.783938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.780327    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-data" pod "186ca244-2b0b-4de3-8b55-2834da4e839a" (UID: "186ca244-2b0b-4de3-8b55-2834da4e839a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:57.783938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.780567    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186ca244-2b0b-4de3-8b55-2834da4e839a-istiod-ca-cert" pod "186ca244-2b0b-4de3-8b55-2834da4e839a" (UID: "186ca244-2b0b-4de3-8b55-2834da4e839a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:39:57.783938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.780828    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-credential-socket" pod "186ca244-2b0b-4de3-8b55-2834da4e839a" (UID: "186ca244-2b0b-4de3-8b55-2834da4e839a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:57.783938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.781129    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-socket" pod "186ca244-2b0b-4de3-8b55-2834da4e839a" (UID: "186ca244-2b0b-4de3-8b55-2834da4e839a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:57.783938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.781332    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-certs" pod "186ca244-2b0b-4de3-8b55-2834da4e839a" (UID: "186ca244-2b0b-4de3-8b55-2834da4e839a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:57.783938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.783744    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-kube-api-access-h7dqk" pod "186ca244-2b0b-4de3-8b55-2834da4e839a" (UID: "186ca244-2b0b-4de3-8b55-2834da4e839a"). InnerVolumeSpecName "kube-api-access-h7dqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:57.784460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.784422    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-podinfo" pod "186ca244-2b0b-4de3-8b55-2834da4e839a" (UID: "186ca244-2b0b-4de3-8b55-2834da4e839a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:39:57.784568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.784513    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-envoy" pod "186ca244-2b0b-4de3-8b55-2834da4e839a" (UID: "186ca244-2b0b-4de3-8b55-2834da4e839a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:39:57.786158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.785897    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-token" pod "186ca244-2b0b-4de3-8b55-2834da4e839a" (UID: "186ca244-2b0b-4de3-8b55-2834da4e839a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:39:57.810837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.810803    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:57.857454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.857424    5420 generic.go:356] "Generic (PLEG): container finished" podID="186ca244-2b0b-4de3-8b55-2834da4e839a" containerID="6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4" exitCode=0
Nov 03 13:39:57.857616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.857466    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"
Nov 03 13:39:57.857616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.857517    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc" event={"ID":"186ca244-2b0b-4de3-8b55-2834da4e839a","Type":"ContainerDied","Data":"6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4"}
Nov 03 13:39:57.857616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.857580    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc" event={"ID":"186ca244-2b0b-4de3-8b55-2834da4e839a","Type":"ContainerDied","Data":"89b25a5518a72de946245de0b00a9b37b2a92af453680b4e98ecfe86c567b099"}
Nov 03 13:39:57.857616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.857611    5420 scope.go:122] "RemoveContainer" containerID="6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4"
Nov 03 13:39:57.871482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.871463    5420 scope.go:122] "RemoveContainer" containerID="6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4"
Nov 03 13:39:57.872105 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:39:57.872053    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4\": container with ID starting with 6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4 not found: ID does not exist" containerID="6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4"
Nov 03 13:39:57.872229 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.872109    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4"} err="failed to get container status \"6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4\": rpc error: code = NotFound desc = could not find container \"6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4\": container with ID starting with 6139c9b585d960771a8b0f6a1191ec1266b514eebb91d68788d61f82bde5e9c4 not found: ID does not exist"
Nov 03 13:39:57.881332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.881310    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:57.881332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.881333    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:57.881461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.881343    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186ca244-2b0b-4de3-8b55-2834da4e839a-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:57.881461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.881352    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:57.881461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.881360    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:57.881461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.881368    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7dqk\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-kube-api-access-h7dqk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:57.881461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.881377    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:57.881461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.881387    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:57.881461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.881395    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186ca244-2b0b-4de3-8b55-2834da4e839a-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:39:57.885891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.885870    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"]
Nov 03 13:39:57.891830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.891810    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-cd80b3-openshift-default-8495b6ff6f-54vxc"]
Nov 03 13:39:57.970720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:57.970695    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:58.535566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:58.535530    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:59.065236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:59.064345    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:59.099032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:59.099000    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186ca244-2b0b-4de3-8b55-2834da4e839a" path="/var/lib/kubelet/pods/186ca244-2b0b-4de3-8b55-2834da4e839a/volumes"
Nov 03 13:39:59.690898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:59.690864    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:39:59.709947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:59.709923    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:59.876558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:39:59.876528    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:00.746439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:00.746400    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:01.340721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.340665    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:40:01.431126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.431090    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-podinfo\") pod \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") "
Nov 03 13:40:01.431339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.431255    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-envoy\") pod \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") "
Nov 03 13:40:01.431339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.431282    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-data\") pod \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") "
Nov 03 13:40:01.431460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.431378    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istiod-ca-cert\") pod \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") "
Nov 03 13:40:01.431460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.431431    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-certs\") pod \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") "
Nov 03 13:40:01.431583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.431489    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-kube-api-access-nhnm5\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-kube-api-access-nhnm5\") pod \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") "
Nov 03 13:40:01.431941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.431663    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-data" pod "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" (UID: "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.431941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.431769    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-certs" pod "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" (UID: "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.431941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.431824    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istiod-ca-cert" pod "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" (UID: "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:01.432745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.432718    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-credential-socket\") pod \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") "
Nov 03 13:40:01.432848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.432761    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-socket\") pod \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") "
Nov 03 13:40:01.432848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.432797    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-token\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-token\") pod \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\" (UID: \"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1\") "
Nov 03 13:40:01.433789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.433509    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-credential-socket" pod "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" (UID: "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.433789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.433521    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-socket" pod "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" (UID: "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.434275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.434248    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-kube-api-access-nhnm5" pod "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" (UID: "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1"). InnerVolumeSpecName "kube-api-access-nhnm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:01.434275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.434007    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-envoy" pod "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" (UID: "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:01.434470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.433609    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.434470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.434014    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-podinfo" pod "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" (UID: "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:40:01.434470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.434310    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.434470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.434327    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.434470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.434343    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.435610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.435585    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-token" pod "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" (UID: "e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:01.527139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.527063    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:01.546476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.546444    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.546476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.546469    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.546476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.546480    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhnm5\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-kube-api-access-nhnm5\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.546773 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.546491    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.546773 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.546501    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:01.894652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.894618    5420 generic.go:356] "Generic (PLEG): container finished" podID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerID="8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2" exitCode=0
Nov 03 13:40:01.895114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.894669    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"
Nov 03 13:40:01.895114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.894703    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" event={"ID":"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1","Type":"ContainerDied","Data":"8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2"}
Nov 03 13:40:01.895114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.894768    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz" event={"ID":"e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1","Type":"ContainerDied","Data":"ae11ed494a894ccab9fb9ee3d92b00eb869c23ce019f824985131cc60c570568"}
Nov 03 13:40:01.895114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.894801    5420 scope.go:122] "RemoveContainer" containerID="8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2"
Nov 03 13:40:01.906491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.906473    5420 scope.go:122] "RemoveContainer" containerID="8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2"
Nov 03 13:40:01.906749 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:40:01.906729    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2\": container with ID starting with 8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2 not found: ID does not exist" containerID="8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2"
Nov 03 13:40:01.906821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.906760    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2"} err="failed to get container status \"8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2\": rpc error: code = NotFound desc = could not find container \"8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2\": container with ID starting with 8339e44e8930d8f761102b8d9170adfd47ed7a8874a023f10d12331666f37ac2 not found: ID does not exist"
Nov 03 13:40:01.921448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.921425    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"]
Nov 03 13:40:01.930772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:01.930752    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-f7dhz"]
Nov 03 13:40:02.125290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:02.124334    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:02.557910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:02.557882    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:02.728122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:02.728086    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:03.086123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:03.086090    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" path="/var/lib/kubelet/pods/e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1/volumes"
Nov 03 13:40:03.326231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:03.322704    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:04.037469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:04.037423    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:04.714020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:04.713829    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:05.343234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:05.342705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:06.092513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:06.092465    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:06.801434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:06.801402    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:07.432183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:07.426737    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:08.036660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:08.036622    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:08.037630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:08.037592    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"]
Nov 03 13:40:08.044282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:08.044251    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"]
Nov 03 13:40:08.049538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:08.049510    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b573dc-nzdd7"]
Nov 03 13:40:08.053760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:08.053740    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae821f-b2kwr"]
Nov 03 13:40:08.607375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:08.607248    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:09.085698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:09.085668    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce7938f-84e4-4a81-b519-b100584149a3" path="/var/lib/kubelet/pods/0ce7938f-84e4-4a81-b519-b100584149a3/volumes"
Nov 03 13:40:09.086123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:09.086109    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2277d95-deaa-457e-aa15-5351aa31d91d" path="/var/lib/kubelet/pods/b2277d95-deaa-457e-aa15-5351aa31d91d/volumes"
Nov 03 13:40:09.267408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:09.267369    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:09.914905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:09.914879    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:10.517206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:10.517158    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:11.109346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:11.107794    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:11.709919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:11.709879    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:12.400245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:12.395590    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:13.006283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:13.005518    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:13.618997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:13.618963    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:14.319890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:14.319854    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:14.925721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:14.925687    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:15.529258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:15.529190    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:16.286550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:16.286513    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:16.932185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:16.932151    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:17.705237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:17.701968    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:18.294643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:18.294605    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:18.896659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:18.896574    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:19.033905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:19.033874    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"]
Nov 03 13:40:19.047548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:19.047513    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fa54a0-r4nxc"]
Nov 03 13:40:19.088069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:19.088025    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08a2cce-fd8a-4a32-9955-ebd5764e4814" path="/var/lib/kubelet/pods/f08a2cce-fd8a-4a32-9955-ebd5764e4814/volumes"
Nov 03 13:40:19.130240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:19.130199    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:19.273396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:19.273367    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:19.441281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:19.441235    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:19.592512 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:19.592307    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:20.181235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:20.174519    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:20.782904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:20.782805    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:21.424294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:21.421786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:22.015458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:22.015290    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:22.370476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:22.370383    5420 scope.go:122] "RemoveContainer" containerID="75a09119aa24c04529c2f5f9c03543fdf1ba33f9916875d0a1753e4df986b910"
Nov 03 13:40:22.693509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:22.693424    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:23.296690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:23.296610    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:24.224418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:24.224374    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:24.732190 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:40:24.732152    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:40:24.969769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:24.969271    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:25.583318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:25.583288    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:26.204484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:26.204453    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:26.798390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:26.798136    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:27.393099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:27.393047    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:28.029013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:28.028817    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:28.611178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:28.611147    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:29.214951 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:29.214920    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:29.780294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:29.779593    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:29.959640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:29.959612    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:30.270333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:30.270303    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:30.641176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:30.641088    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:31.086255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:31.086229    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"]
Nov 03 13:40:31.110525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:31.110498    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a137e9-plwcp"]
Nov 03 13:40:31.579508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:31.579478    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:32.223958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:32.223911    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:32.922289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:32.922248    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:33.085199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:33.085167    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0972fa32-c3eb-4522-89c1-700fbc38f167" path="/var/lib/kubelet/pods/0972fa32-c3eb-4522-89c1-700fbc38f167/volumes"
Nov 03 13:40:33.485006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:33.483880    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:34.116837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:34.116801    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:34.820675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:34.816920    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:35.429846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:35.429815    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:36.028769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:36.028725    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:36.272244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:36.272185    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:36.499745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:36.499715    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:36.918099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:36.918007    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:37.504828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:37.503286    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:38.227169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:38.227128    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:38.790074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:38.790042    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:39.365441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:39.365406    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:40.025649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:40.025617    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:40.692821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:40.692789    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:41.194233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:41.192879    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:41.788618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:41.788565    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:42.430727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:42.430690    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:43.022132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:43.022080    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:43.663325 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:43.651739    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:43.966554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:43.966520    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:44.114791 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:44.114760    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:44.419883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:44.419770    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:44.963443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:44.963403    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:40:45.009549 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:45.009516    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:45.028858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:45.028814    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"]
Nov 03 13:40:45.034458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:45.034417    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc66dcf4-nq6cr"]
Nov 03 13:40:45.085500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:45.085474    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3497c5-1eea-4841-9a2e-c64ccf7e571b" path="/var/lib/kubelet/pods/6a3497c5-1eea-4841-9a2e-c64ccf7e571b/volumes"
Nov 03 13:40:45.633181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:45.633145    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:46.112880 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:46.112842    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:47.395160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:47.395125    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:48.131351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:48.131285    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:49.028116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:49.028086    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:49.791659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:49.791626    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:50.225263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:50.225126    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:50.374509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:50.374478    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:50.530129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:50.530024    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:50.703605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:50.703559    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:50.926374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:50.926298    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"]
Nov 03 13:40:50.926562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:50.926535    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy" containerID="cri-o://4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc" gracePeriod=30
Nov 03 13:40:51.027232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:51.023201    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:51.342353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:51.342321    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:51.580145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:51.580107    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:51.816956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:51.816927    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:52.267874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:52.267839    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:52.884186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:52.884157    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:53.600877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:53.600833    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:54.407268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.407235    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:54.552205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.552174    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"]
Nov 03 13:40:54.552852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.552838    5420 cpu_manager.go:414] "containerMap: removing container" podUID="89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" containerName="payload-pre-processing"
Nov 03 13:40:54.552898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.552854    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" containerName="payload-pre-processing"
Nov 03 13:40:54.552898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.552880    5420 cpu_manager.go:414] "containerMap: removing container" podUID="186ca244-2b0b-4de3-8b55-2834da4e839a" containerName="istio-proxy"
Nov 03 13:40:54.552898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.552885    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186ca244-2b0b-4de3-8b55-2834da4e839a" containerName="istio-proxy"
Nov 03 13:40:54.552898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.552894    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy"
Nov 03 13:40:54.552898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.552899    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy"
Nov 03 13:40:54.553063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.553004    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="186ca244-2b0b-4de3-8b55-2834da4e839a" containerName="istio-proxy"
Nov 03 13:40:54.553063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.553015    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3beb4bb-7c9f-4b98-a1fd-4457bd6225a1" containerName="istio-proxy"
Nov 03 13:40:54.553063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.553026    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="89bd0a5a-58c6-4bd6-b731-c2c7743c4b65" containerName="payload-pre-processing"
Nov 03 13:40:54.556886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.556871    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.562509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.562487    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:40:54.562668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.562535    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:40:54.563705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.563681    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:40:54.563861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.563712    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:40:54.565986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.565967    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"]
Nov 03 13:40:54.622418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.622387    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee95c914-wxbq7\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.622567 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.622524    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee95c914-wxbq7\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.622688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.622673    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f22d6\" (UniqueName: \"kubernetes.io/projected/1cebddea-d8a9-4939-a348-893d86da6489-kube-api-access-f22d6\") pod \"maas-api-revoke-sub-ee95c914-wxbq7\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.724353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.724322    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee95c914-wxbq7\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.724557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.724536    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee95c914-wxbq7\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.724716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.724701    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f22d6\" (UniqueName: \"kubernetes.io/projected/1cebddea-d8a9-4939-a348-893d86da6489-kube-api-access-f22d6\") pod \"maas-api-revoke-sub-ee95c914-wxbq7\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.725495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.725470    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee95c914-wxbq7\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.726796 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.726769    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee95c914-wxbq7\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.733040 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:40:54.733010    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:40:54.734028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.734008    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f22d6\" (UniqueName: \"kubernetes.io/projected/1cebddea-d8a9-4939-a348-893d86da6489-kube-api-access-f22d6\") pod \"maas-api-revoke-sub-ee95c914-wxbq7\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:54.872371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:54.872345    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:55.019153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.018966    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:55.156460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.156434    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"]
Nov 03 13:40:55.156710 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:40:55.156690    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cebddea_d8a9_4939_a348_893d86da6489.slice/crio-9e947d892b6834f7bbba8da8c4ce35a639169e91e9d99a1f3c67992474d12019 WatchSource:0}: Error finding container 9e947d892b6834f7bbba8da8c4ce35a639169e91e9d99a1f3c67992474d12019: Status 404 returned error can't find the container with id 9e947d892b6834f7bbba8da8c4ce35a639169e91e9d99a1f3c67992474d12019
Nov 03 13:40:55.201643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.201617    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"]
Nov 03 13:40:55.206892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.206876    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.214424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.214398    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"]
Nov 03 13:40:55.334724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.334644    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bafc04d1-w569l\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.334889 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.334788    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvj7n\" (UniqueName: \"kubernetes.io/projected/32cfed13-1df9-4a89-985e-db71b0e6f87f-kube-api-access-vvj7n\") pod \"maas-api-revoke-sub-bafc04d1-w569l\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.334941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.334901    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bafc04d1-w569l\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.405418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.405387    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7" event={"ID":"1cebddea-d8a9-4939-a348-893d86da6489","Type":"ContainerStarted","Data":"9e947d892b6834f7bbba8da8c4ce35a639169e91e9d99a1f3c67992474d12019"}
Nov 03 13:40:55.436355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.436330    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bafc04d1-w569l\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.436779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.436467    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvj7n\" (UniqueName: \"kubernetes.io/projected/32cfed13-1df9-4a89-985e-db71b0e6f87f-kube-api-access-vvj7n\") pod \"maas-api-revoke-sub-bafc04d1-w569l\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.436779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.436593    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bafc04d1-w569l\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.436940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.436922    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bafc04d1-w569l\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.439186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.439164    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bafc04d1-w569l\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.444788 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.444770    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvj7n\" (UniqueName: \"kubernetes.io/projected/32cfed13-1df9-4a89-985e-db71b0e6f87f-kube-api-access-vvj7n\") pod \"maas-api-revoke-sub-bafc04d1-w569l\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.524453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.524412    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:55.775633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.773772    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:55.809531 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:40:55.809501    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cfed13_1df9_4a89_985e_db71b0e6f87f.slice/crio-11d028dc5267793b8691090b305ca1b315634f2f6bc5e81b0cd8106573085f3c WatchSource:0}: Error finding container 11d028dc5267793b8691090b305ca1b315634f2f6bc5e81b0cd8106573085f3c: Status 404 returned error can't find the container with id 11d028dc5267793b8691090b305ca1b315634f2f6bc5e81b0cd8106573085f3c
Nov 03 13:40:55.810299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:55.810276    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"]
Nov 03 13:40:56.230224 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.230194    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:40:56.366071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.365970    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-data\") pod \"d1f6377b-b159-4b6c-85c6-ad75b000c640\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") "
Nov 03 13:40:56.366071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366020    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-certs\") pod \"d1f6377b-b159-4b6c-85c6-ad75b000c640\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") "
Nov 03 13:40:56.366071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366058    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-credential-socket\") pod \"d1f6377b-b159-4b6c-85c6-ad75b000c640\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") "
Nov 03 13:40:56.366415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366125    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-podinfo\") pod \"d1f6377b-b159-4b6c-85c6-ad75b000c640\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") "
Nov 03 13:40:56.366415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366163    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-envoy\") pod \"d1f6377b-b159-4b6c-85c6-ad75b000c640\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") "
Nov 03 13:40:56.366415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366272    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-data" pod "d1f6377b-b159-4b6c-85c6-ad75b000c640" (UID: "d1f6377b-b159-4b6c-85c6-ad75b000c640"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:56.366415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366304    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f6377b-b159-4b6c-85c6-ad75b000c640-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f6377b-b159-4b6c-85c6-ad75b000c640-istiod-ca-cert\") pod \"d1f6377b-b159-4b6c-85c6-ad75b000c640\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") "
Nov 03 13:40:56.366415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366346    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-socket\") pod \"d1f6377b-b159-4b6c-85c6-ad75b000c640\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") "
Nov 03 13:40:56.366415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366351    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-credential-socket" pod "d1f6377b-b159-4b6c-85c6-ad75b000c640" (UID: "d1f6377b-b159-4b6c-85c6-ad75b000c640"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:56.366732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366415    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-certs" pod "d1f6377b-b159-4b6c-85c6-ad75b000c640" (UID: "d1f6377b-b159-4b6c-85c6-ad75b000c640"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:56.366732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366449    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-kube-api-access-5m5c4\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-kube-api-access-5m5c4\") pod \"d1f6377b-b159-4b6c-85c6-ad75b000c640\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") "
Nov 03 13:40:56.366732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366485    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-token\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-token\") pod \"d1f6377b-b159-4b6c-85c6-ad75b000c640\" (UID: \"d1f6377b-b159-4b6c-85c6-ad75b000c640\") "
Nov 03 13:40:56.366732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366634    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-socket" pod "d1f6377b-b159-4b6c-85c6-ad75b000c640" (UID: "d1f6377b-b159-4b6c-85c6-ad75b000c640"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:56.366732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.366696    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f6377b-b159-4b6c-85c6-ad75b000c640-istiod-ca-cert" pod "d1f6377b-b159-4b6c-85c6-ad75b000c640" (UID: "d1f6377b-b159-4b6c-85c6-ad75b000c640"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:56.367330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.367306    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:56.367443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.367336    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:56.367443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.367353    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f6377b-b159-4b6c-85c6-ad75b000c640-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:56.367443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.367367    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:56.367443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.367381    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:56.368445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.368420    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-envoy" pod "d1f6377b-b159-4b6c-85c6-ad75b000c640" (UID: "d1f6377b-b159-4b6c-85c6-ad75b000c640"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:56.368544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.368517    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-podinfo" pod "d1f6377b-b159-4b6c-85c6-ad75b000c640" (UID: "d1f6377b-b159-4b6c-85c6-ad75b000c640"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:40:56.368597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.368583    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-kube-api-access-5m5c4" pod "d1f6377b-b159-4b6c-85c6-ad75b000c640" (UID: "d1f6377b-b159-4b6c-85c6-ad75b000c640"). InnerVolumeSpecName "kube-api-access-5m5c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:56.368664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.368645    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-token" pod "d1f6377b-b159-4b6c-85c6-ad75b000c640" (UID: "d1f6377b-b159-4b6c-85c6-ad75b000c640"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:56.425496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.421482    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:56.436085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.436026    5420 generic.go:356] "Generic (PLEG): container finished" podID="1cebddea-d8a9-4939-a348-893d86da6489" containerID="d90f8f9b2d0c5ae2b19c913bb6195308c227cf13ab5aac981c990c63dbdd1ab4" exitCode=0
Nov 03 13:40:56.436322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.436276    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7" event={"ID":"1cebddea-d8a9-4939-a348-893d86da6489","Type":"ContainerDied","Data":"d90f8f9b2d0c5ae2b19c913bb6195308c227cf13ab5aac981c990c63dbdd1ab4"}
Nov 03 13:40:56.440227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.438540    5420 generic.go:356] "Generic (PLEG): container finished" podID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerID="4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc" exitCode=0
Nov 03 13:40:56.440227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.438751    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" event={"ID":"d1f6377b-b159-4b6c-85c6-ad75b000c640","Type":"ContainerDied","Data":"4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc"}
Nov 03 13:40:56.440227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.438799    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6" event={"ID":"d1f6377b-b159-4b6c-85c6-ad75b000c640","Type":"ContainerDied","Data":"fb53e608fda3ddab71b26ca291744cf01506e2928fe9e46f87f3d14fabd0bf6c"}
Nov 03 13:40:56.440227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.438838    5420 scope.go:122] "RemoveContainer" containerID="4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc"
Nov 03 13:40:56.440227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.439007    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"
Nov 03 13:40:56.471468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.470994    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:56.471468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.471027    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:56.471468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.471043    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m5c4\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-kube-api-access-5m5c4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:56.471468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.471058    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f6377b-b159-4b6c-85c6-ad75b000c640-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:56.475288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.475194    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l" event={"ID":"32cfed13-1df9-4a89-985e-db71b0e6f87f","Type":"ContainerStarted","Data":"d707a4587f7a220d183386ee44213e007dd8cf220935143acf1f43d1adca483c"}
Nov 03 13:40:56.475416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.475385    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l" event={"ID":"32cfed13-1df9-4a89-985e-db71b0e6f87f","Type":"ContainerStarted","Data":"11d028dc5267793b8691090b305ca1b315634f2f6bc5e81b0cd8106573085f3c"}
Nov 03 13:40:56.486695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.486675    5420 scope.go:122] "RemoveContainer" containerID="4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc"
Nov 03 13:40:56.486971 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:40:56.486945    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc\": container with ID starting with 4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc not found: ID does not exist" containerID="4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc"
Nov 03 13:40:56.487034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.486979    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc"} err="failed to get container status \"4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc\": rpc error: code = NotFound desc = could not find container \"4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc\": container with ID starting with 4b4a635aa3202660fd865ea16043e0fb4837a1d1bd3f45fbbe42c9ecce19cbbc not found: ID does not exist"
Nov 03 13:40:56.504175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.504137    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l" podStartSLOduration=1.504125094 podStartE2EDuration="1.504125094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:56.50032482 +0000 UTC m=+1897.578089845" watchObservedRunningTime="2026-11-03 13:40:56.504125094 +0000 UTC m=+1897.581890119"
Nov 03 13:40:56.519966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.519942    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"]
Nov 03 13:40:56.524352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:56.524332    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-z94q6"]
Nov 03 13:40:57.085272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.085191    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" path="/var/lib/kubelet/pods/d1f6377b-b159-4b6c-85c6-ad75b000c640/volumes"
Nov 03 13:40:57.129856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.129822    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:57.485957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.485926    5420 generic.go:356] "Generic (PLEG): container finished" podID="32cfed13-1df9-4a89-985e-db71b0e6f87f" containerID="d707a4587f7a220d183386ee44213e007dd8cf220935143acf1f43d1adca483c" exitCode=0
Nov 03 13:40:57.486451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.486015    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l" event={"ID":"32cfed13-1df9-4a89-985e-db71b0e6f87f","Type":"ContainerDied","Data":"d707a4587f7a220d183386ee44213e007dd8cf220935143acf1f43d1adca483c"}
Nov 03 13:40:57.659864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.659842    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:57.733558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.732237    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:57.806840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.806750    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-xks-ca\") pod \"1cebddea-d8a9-4939-a348-893d86da6489\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") "
Nov 03 13:40:57.807027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.806873    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-openshift-ca\") pod \"1cebddea-d8a9-4939-a348-893d86da6489\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") "
Nov 03 13:40:57.807027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.806897    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cebddea-d8a9-4939-a348-893d86da6489-kube-api-access-f22d6\" (UniqueName: \"kubernetes.io/projected/1cebddea-d8a9-4939-a348-893d86da6489-kube-api-access-f22d6\") pod \"1cebddea-d8a9-4939-a348-893d86da6489\" (UID: \"1cebddea-d8a9-4939-a348-893d86da6489\") "
Nov 03 13:40:57.807332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.807308    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-openshift-ca" pod "1cebddea-d8a9-4939-a348-893d86da6489" (UID: "1cebddea-d8a9-4939-a348-893d86da6489"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:57.807679 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.807659    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:57.809370 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.809347    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-xks-ca" pod "1cebddea-d8a9-4939-a348-893d86da6489" (UID: "1cebddea-d8a9-4939-a348-893d86da6489"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:40:57.809525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.809397    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cebddea-d8a9-4939-a348-893d86da6489-kube-api-access-f22d6" pod "1cebddea-d8a9-4939-a348-893d86da6489" (UID: "1cebddea-d8a9-4939-a348-893d86da6489"). InnerVolumeSpecName "kube-api-access-f22d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:57.908733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.908706    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cebddea-d8a9-4939-a348-893d86da6489-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:57.908733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:57.908731    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f22d6\" (UniqueName: \"kubernetes.io/projected/1cebddea-d8a9-4939-a348-893d86da6489-kube-api-access-f22d6\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:58.040337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.040308    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:58.198079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.197995    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:58.360235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.356721    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:58.509147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.509108    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"
Nov 03 13:40:58.514328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.514290    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7" event={"ID":"1cebddea-d8a9-4939-a348-893d86da6489","Type":"ContainerDied","Data":"9e947d892b6834f7bbba8da8c4ce35a639169e91e9d99a1f3c67992474d12019"}
Nov 03 13:40:58.514520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.514361    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e947d892b6834f7bbba8da8c4ce35a639169e91e9d99a1f3c67992474d12019"
Nov 03 13:40:58.574884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.574850    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:58.720861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.720787    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:58.765400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.765345    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:58.830887 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.830851    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cfed13-1df9-4a89-985e-db71b0e6f87f-kube-api-access-vvj7n\" (UniqueName: \"kubernetes.io/projected/32cfed13-1df9-4a89-985e-db71b0e6f87f-kube-api-access-vvj7n\") pod \"32cfed13-1df9-4a89-985e-db71b0e6f87f\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") "
Nov 03 13:40:58.831080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.830933    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-openshift-ca\") pod \"32cfed13-1df9-4a89-985e-db71b0e6f87f\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") "
Nov 03 13:40:58.831139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.831113    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-xks-ca\") pod \"32cfed13-1df9-4a89-985e-db71b0e6f87f\" (UID: \"32cfed13-1df9-4a89-985e-db71b0e6f87f\") "
Nov 03 13:40:58.831683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.831652    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-openshift-ca" pod "32cfed13-1df9-4a89-985e-db71b0e6f87f" (UID: "32cfed13-1df9-4a89-985e-db71b0e6f87f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:40:58.831828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.831756    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:58.833425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.833403    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-xks-ca" pod "32cfed13-1df9-4a89-985e-db71b0e6f87f" (UID: "32cfed13-1df9-4a89-985e-db71b0e6f87f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:40:58.833556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.833541    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cfed13-1df9-4a89-985e-db71b0e6f87f-kube-api-access-vvj7n" pod "32cfed13-1df9-4a89-985e-db71b0e6f87f" (UID: "32cfed13-1df9-4a89-985e-db71b0e6f87f"). InnerVolumeSpecName "kube-api-access-vvj7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:58.933693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.933662    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cfed13-1df9-4a89-985e-db71b0e6f87f-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:58.933693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:58.933688    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvj7n\" (UniqueName: \"kubernetes.io/projected/32cfed13-1df9-4a89-985e-db71b0e6f87f-kube-api-access-vvj7n\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:59.311016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:59.310530    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:59.517573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:59.517534    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l" event={"ID":"32cfed13-1df9-4a89-985e-db71b0e6f87f","Type":"ContainerDied","Data":"11d028dc5267793b8691090b305ca1b315634f2f6bc5e81b0cd8106573085f3c"}
Nov 03 13:40:59.517977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:59.517589    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11d028dc5267793b8691090b305ca1b315634f2f6bc5e81b0cd8106573085f3c"
Nov 03 13:40:59.517977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:59.517546    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"
Nov 03 13:40:59.870008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:40:59.869935    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:00.278201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:00.278161    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:00.897235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:00.896791    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:01.548447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:01.548417    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:02.233618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:02.233573    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:02.731277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:02.730740    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:03.337625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:03.337587    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:03.579195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:03.579158    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:03.732800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:03.732768    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:03.993289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:03.991225    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:04.479068 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.479038    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"]
Nov 03 13:41:04.479614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.479253    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy" containerID="cri-o://ea580a6504a2c949bfa62913b2d572d861726456f18ea43ebd1a9146d60e5b13" gracePeriod=30
Nov 03 13:41:04.756535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.755778    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:04.756535 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.755971    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"]
Nov 03 13:41:04.761674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.756981    5420 cpu_manager.go:414] "containerMap: removing container" podUID="1cebddea-d8a9-4939-a348-893d86da6489" containerName="revoke-keys"
Nov 03 13:41:04.761674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.757006    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cebddea-d8a9-4939-a348-893d86da6489" containerName="revoke-keys"
Nov 03 13:41:04.761674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.757065    5420 cpu_manager.go:414] "containerMap: removing container" podUID="32cfed13-1df9-4a89-985e-db71b0e6f87f" containerName="revoke-keys"
Nov 03 13:41:04.761674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.757074    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cfed13-1df9-4a89-985e-db71b0e6f87f" containerName="revoke-keys"
Nov 03 13:41:04.761674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.757100    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy"
Nov 03 13:41:04.761674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.757107    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy"
Nov 03 13:41:04.761674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.757311    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cfed13-1df9-4a89-985e-db71b0e6f87f" containerName="revoke-keys"
Nov 03 13:41:04.761674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.757337    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f6377b-b159-4b6c-85c6-ad75b000c640" containerName="istio-proxy"
Nov 03 13:41:04.761674 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.757366    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cebddea-d8a9-4939-a348-893d86da6489" containerName="revoke-keys"
Nov 03 13:41:04.763948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.763924    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"]
Nov 03 13:41:04.764296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.764278    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:04.772301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.769350    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:41:04.772301 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.769938    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:41:04.824079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.824049    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:41:04.824304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.824159    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:41:04.852313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.852276    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-487ba6f7-rp9jf\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:04.852491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.852393    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slpjn\" (UniqueName: \"kubernetes.io/projected/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-kube-api-access-slpjn\") pod \"maas-api-revoke-sub-487ba6f7-rp9jf\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:04.852536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.852508    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-487ba6f7-rp9jf\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:04.953841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.953800    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slpjn\" (UniqueName: \"kubernetes.io/projected/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-kube-api-access-slpjn\") pod \"maas-api-revoke-sub-487ba6f7-rp9jf\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:04.954030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.953921    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-487ba6f7-rp9jf\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:04.954191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.954169    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-487ba6f7-rp9jf\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:04.954870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.954839    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-487ba6f7-rp9jf\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:04.956327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.956306    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-487ba6f7-rp9jf\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:04.962656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:04.962635    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slpjn\" (UniqueName: \"kubernetes.io/projected/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-kube-api-access-slpjn\") pod \"maas-api-revoke-sub-487ba6f7-rp9jf\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:05.093940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.093853    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:05.376822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.376792    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"]
Nov 03 13:41:05.377144 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:41:05.377109    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd410728a_5f1d_4fd2_a5a7_c4d9e2e0222d.slice/crio-266c9b0a44aa5803475c778857a3bc14fedbe15d08b356ab060cefb3bd4f2507 WatchSource:0}: Error finding container 266c9b0a44aa5803475c778857a3bc14fedbe15d08b356ab060cefb3bd4f2507: Status 404 returned error can't find the container with id 266c9b0a44aa5803475c778857a3bc14fedbe15d08b356ab060cefb3bd4f2507
Nov 03 13:41:05.426599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.426567    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"]
Nov 03 13:41:05.432685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.432658    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.439385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.439357    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"]
Nov 03 13:41:05.461182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.461149    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ths4\" (UniqueName: \"kubernetes.io/projected/a29f33dc-fd1f-40fc-8779-034d313df505-kube-api-access-8ths4\") pod \"maas-api-revoke-sub-4c6857e6-sf28k\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.461419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.461404    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6857e6-sf28k\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.461488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.461464    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6857e6-sf28k\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.537517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.536101    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:05.564267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.562970    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ths4\" (UniqueName: \"kubernetes.io/projected/a29f33dc-fd1f-40fc-8779-034d313df505-kube-api-access-8ths4\") pod \"maas-api-revoke-sub-4c6857e6-sf28k\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.564267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.563330    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6857e6-sf28k\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.564267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.563424    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6857e6-sf28k\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.564627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.564330    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6857e6-sf28k\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.566983 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.566958    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6857e6-sf28k\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.573725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.573701    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ths4\" (UniqueName: \"kubernetes.io/projected/a29f33dc-fd1f-40fc-8779-034d313df505-kube-api-access-8ths4\") pod \"maas-api-revoke-sub-4c6857e6-sf28k\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.585892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.585863    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf" event={"ID":"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d","Type":"ContainerStarted","Data":"266c9b0a44aa5803475c778857a3bc14fedbe15d08b356ab060cefb3bd4f2507"}
Nov 03 13:41:05.751380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.751350    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:05.843352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.841805    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:05.989006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:05.988925    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:06.037688 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:41:06.037656    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29f33dc_fd1f_40fc_8779_034d313df505.slice/crio-9118b7b0bf12fc11298060b28a6914eaf1a25f20a149cf87f9f6d587a30e8da3 WatchSource:0}: Error finding container 9118b7b0bf12fc11298060b28a6914eaf1a25f20a149cf87f9f6d587a30e8da3: Status 404 returned error can't find the container with id 9118b7b0bf12fc11298060b28a6914eaf1a25f20a149cf87f9f6d587a30e8da3
Nov 03 13:41:06.037875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.037854    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"]
Nov 03 13:41:06.229350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.229313    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:06.388883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.388852    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:06.598027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.597946    5420 generic.go:356] "Generic (PLEG): container finished" podID="d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d" containerID="8db4fab932c181e5e9af152957b373f1212c8acfda52fceaddb0129e90bb51e4" exitCode=0
Nov 03 13:41:06.598597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.598102    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf" event={"ID":"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d","Type":"ContainerDied","Data":"8db4fab932c181e5e9af152957b373f1212c8acfda52fceaddb0129e90bb51e4"}
Nov 03 13:41:06.618988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.618825    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k" event={"ID":"a29f33dc-fd1f-40fc-8779-034d313df505","Type":"ContainerStarted","Data":"34df9cc96f22e11fdabe90775c7214f17ef1d7bfa565b34f62a1c71fda4917b2"}
Nov 03 13:41:06.618988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.618882    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k" event={"ID":"a29f33dc-fd1f-40fc-8779-034d313df505","Type":"ContainerStarted","Data":"9118b7b0bf12fc11298060b28a6914eaf1a25f20a149cf87f9f6d587a30e8da3"}
Nov 03 13:41:06.644997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.644964    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"]
Nov 03 13:41:06.645193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.645174    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy" containerID="cri-o://ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8" gracePeriod=30
Nov 03 13:41:06.660311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.655465    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k" podStartSLOduration=1.655451945 podStartE2EDuration="1.655451945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:06.651590684 +0000 UTC m=+1907.729355711" watchObservedRunningTime="2026-11-03 13:41:06.655451945 +0000 UTC m=+1907.733216970"
Nov 03 13:41:06.721569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:06.721532    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:07.056826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.056796    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:07.195047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.195014    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:07.478814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.478784    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"]
Nov 03 13:41:07.483838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.483815    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.491061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.491035    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"]
Nov 03 13:41:07.602036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.602000    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205bbc4e-tt9bb\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.602422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.602244    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9sll\" (UniqueName: \"kubernetes.io/projected/94de26d1-fcbf-460d-aa5b-821b4647cb7a-kube-api-access-t9sll\") pod \"maas-api-revoke-sub-205bbc4e-tt9bb\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.602422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.602301    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205bbc4e-tt9bb\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.628396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.628364    5420 generic.go:356] "Generic (PLEG): container finished" podID="a29f33dc-fd1f-40fc-8779-034d313df505" containerID="34df9cc96f22e11fdabe90775c7214f17ef1d7bfa565b34f62a1c71fda4917b2" exitCode=0
Nov 03 13:41:07.628545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.628446    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k" event={"ID":"a29f33dc-fd1f-40fc-8779-034d313df505","Type":"ContainerDied","Data":"34df9cc96f22e11fdabe90775c7214f17ef1d7bfa565b34f62a1c71fda4917b2"}
Nov 03 13:41:07.709908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.709856    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9sll\" (UniqueName: \"kubernetes.io/projected/94de26d1-fcbf-460d-aa5b-821b4647cb7a-kube-api-access-t9sll\") pod \"maas-api-revoke-sub-205bbc4e-tt9bb\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.710154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.709963    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205bbc4e-tt9bb\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.710154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.710098    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205bbc4e-tt9bb\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.711136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.711106    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205bbc4e-tt9bb\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.735299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.729673    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205bbc4e-tt9bb\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.735299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.729954    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:07.735299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.732308    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9sll\" (UniqueName: \"kubernetes.io/projected/94de26d1-fcbf-460d-aa5b-821b4647cb7a-kube-api-access-t9sll\") pod \"maas-api-revoke-sub-205bbc4e-tt9bb\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.800757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.800724    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:07.848228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.848149    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:07.913288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.913254    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-openshift-ca\") pod \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") "
Nov 03 13:41:07.913462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.913425    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-xks-ca\") pod \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") "
Nov 03 13:41:07.913587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.913546    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-kube-api-access-slpjn\" (UniqueName: \"kubernetes.io/projected/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-kube-api-access-slpjn\") pod \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\" (UID: \"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d\") "
Nov 03 13:41:07.913678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.913640    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-openshift-ca" pod "d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d" (UID: "d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:07.914309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.914285    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:07.916060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.916037    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-xks-ca" pod "d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d" (UID: "d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:07.916166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:07.916089    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-kube-api-access-slpjn" pod "d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d" (UID: "d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d"). InnerVolumeSpecName "kube-api-access-slpjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:08.015297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.015204    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:08.015297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.015245    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slpjn\" (UniqueName: \"kubernetes.io/projected/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d-kube-api-access-slpjn\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:08.088787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.088758    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"]
Nov 03 13:41:08.089886 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:41:08.089859    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94de26d1_fcbf_460d_aa5b_821b4647cb7a.slice/crio-f2f95a83b531fb144e4dd745dfa34f2574c0c8e42db29047aa32b9ce5fe8eaa4 WatchSource:0}: Error finding container f2f95a83b531fb144e4dd745dfa34f2574c0c8e42db29047aa32b9ce5fe8eaa4: Status 404 returned error can't find the container with id f2f95a83b531fb144e4dd745dfa34f2574c0c8e42db29047aa32b9ce5fe8eaa4
Nov 03 13:41:08.290300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.287022    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:08.637701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.637618    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb" event={"ID":"94de26d1-fcbf-460d-aa5b-821b4647cb7a","Type":"ContainerStarted","Data":"1e7cf891ba826827001425ef9d6a92d428b36ee87073217dc618d68d9927660f"}
Nov 03 13:41:08.637701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.637683    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb" event={"ID":"94de26d1-fcbf-460d-aa5b-821b4647cb7a","Type":"ContainerStarted","Data":"f2f95a83b531fb144e4dd745dfa34f2574c0c8e42db29047aa32b9ce5fe8eaa4"}
Nov 03 13:41:08.639163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.639145    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"
Nov 03 13:41:08.639300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.639194    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf" event={"ID":"d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d","Type":"ContainerDied","Data":"266c9b0a44aa5803475c778857a3bc14fedbe15d08b356ab060cefb3bd4f2507"}
Nov 03 13:41:08.639300 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.639259    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="266c9b0a44aa5803475c778857a3bc14fedbe15d08b356ab060cefb3bd4f2507"
Nov 03 13:41:08.660646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.660593    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb" podStartSLOduration=1.6605791170000002 podStartE2EDuration="1.660579117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:08.656046993 +0000 UTC m=+1909.733812011" watchObservedRunningTime="2026-11-03 13:41:08.660579117 +0000 UTC m=+1909.738344179"
Nov 03 13:41:08.797263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.797231    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:08.893350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.877462    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:08.927722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.927690    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-openshift-ca\") pod \"a29f33dc-fd1f-40fc-8779-034d313df505\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") "
Nov 03 13:41:08.927928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.927872    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29f33dc-fd1f-40fc-8779-034d313df505-kube-api-access-8ths4\" (UniqueName: \"kubernetes.io/projected/a29f33dc-fd1f-40fc-8779-034d313df505-kube-api-access-8ths4\") pod \"a29f33dc-fd1f-40fc-8779-034d313df505\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") "
Nov 03 13:41:08.927928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.927915    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-xks-ca\") pod \"a29f33dc-fd1f-40fc-8779-034d313df505\" (UID: \"a29f33dc-fd1f-40fc-8779-034d313df505\") "
Nov 03 13:41:08.928072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.928011    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-openshift-ca" pod "a29f33dc-fd1f-40fc-8779-034d313df505" (UID: "a29f33dc-fd1f-40fc-8779-034d313df505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:08.928831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.928804    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:08.930003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.929981    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29f33dc-fd1f-40fc-8779-034d313df505-kube-api-access-8ths4" pod "a29f33dc-fd1f-40fc-8779-034d313df505" (UID: "a29f33dc-fd1f-40fc-8779-034d313df505"). InnerVolumeSpecName "kube-api-access-8ths4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:08.931490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:08.931459    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-xks-ca" pod "a29f33dc-fd1f-40fc-8779-034d313df505" (UID: "a29f33dc-fd1f-40fc-8779-034d313df505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:09.029774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.029743    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ths4\" (UniqueName: \"kubernetes.io/projected/a29f33dc-fd1f-40fc-8779-034d313df505-kube-api-access-8ths4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.029774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.029771    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29f33dc-fd1f-40fc-8779-034d313df505-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.518603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.518565    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:09.662037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.662007    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k" event={"ID":"a29f33dc-fd1f-40fc-8779-034d313df505","Type":"ContainerDied","Data":"9118b7b0bf12fc11298060b28a6914eaf1a25f20a149cf87f9f6d587a30e8da3"}
Nov 03 13:41:09.662408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.662087    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9118b7b0bf12fc11298060b28a6914eaf1a25f20a149cf87f9f6d587a30e8da3"
Nov 03 13:41:09.662408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.662151    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"
Nov 03 13:41:09.665876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.665843    5420 generic.go:356] "Generic (PLEG): container finished" podID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerID="ea580a6504a2c949bfa62913b2d572d861726456f18ea43ebd1a9146d60e5b13" exitCode=0
Nov 03 13:41:09.666031 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.665876    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" event={"ID":"4ca7984a-88de-4f3e-8b32-ca9f9e15a649","Type":"ContainerDied","Data":"ea580a6504a2c949bfa62913b2d572d861726456f18ea43ebd1a9146d60e5b13"}
Nov 03 13:41:09.669572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.669303    5420 generic.go:356] "Generic (PLEG): container finished" podID="94de26d1-fcbf-460d-aa5b-821b4647cb7a" containerID="1e7cf891ba826827001425ef9d6a92d428b36ee87073217dc618d68d9927660f" exitCode=0
Nov 03 13:41:09.669572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.669354    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb" event={"ID":"94de26d1-fcbf-460d-aa5b-821b4647cb7a","Type":"ContainerDied","Data":"1e7cf891ba826827001425ef9d6a92d428b36ee87073217dc618d68d9927660f"}
Nov 03 13:41:09.776728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.776668    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:41:09.843142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843105    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-certs\") pod \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") "
Nov 03 13:41:09.843386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843197    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-podinfo\") pod \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") "
Nov 03 13:41:09.843386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843266    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-token\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-token\") pod \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") "
Nov 03 13:41:09.843386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843307    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-kube-api-access-6lkgz\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-kube-api-access-6lkgz\") pod \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") "
Nov 03 13:41:09.843386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843386    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-envoy\") pod \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") "
Nov 03 13:41:09.843638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843443    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-certs" pod "4ca7984a-88de-4f3e-8b32-ca9f9e15a649" (UID: "4ca7984a-88de-4f3e-8b32-ca9f9e15a649"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:09.843638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843630    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-socket\") pod \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") "
Nov 03 13:41:09.843750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843682    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-data\") pod \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") "
Nov 03 13:41:09.843750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843727    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-credential-socket\") pod \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") "
Nov 03 13:41:09.843870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843783    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istiod-ca-cert\") pod \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\" (UID: \"4ca7984a-88de-4f3e-8b32-ca9f9e15a649\") "
Nov 03 13:41:09.844024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.843994    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-socket" pod "4ca7984a-88de-4f3e-8b32-ca9f9e15a649" (UID: "4ca7984a-88de-4f3e-8b32-ca9f9e15a649"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:09.844431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.844403    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istiod-ca-cert" pod "4ca7984a-88de-4f3e-8b32-ca9f9e15a649" (UID: "4ca7984a-88de-4f3e-8b32-ca9f9e15a649"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:09.844570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.844518    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.844570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.844541    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.844703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.844668    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-data" pod "4ca7984a-88de-4f3e-8b32-ca9f9e15a649" (UID: "4ca7984a-88de-4f3e-8b32-ca9f9e15a649"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:09.844786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.844763    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-credential-socket" pod "4ca7984a-88de-4f3e-8b32-ca9f9e15a649" (UID: "4ca7984a-88de-4f3e-8b32-ca9f9e15a649"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:09.845997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.845960    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-envoy" pod "4ca7984a-88de-4f3e-8b32-ca9f9e15a649" (UID: "4ca7984a-88de-4f3e-8b32-ca9f9e15a649"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:09.847431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.847403    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-token" pod "4ca7984a-88de-4f3e-8b32-ca9f9e15a649" (UID: "4ca7984a-88de-4f3e-8b32-ca9f9e15a649"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:09.847574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.847549    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-kube-api-access-6lkgz" pod "4ca7984a-88de-4f3e-8b32-ca9f9e15a649" (UID: "4ca7984a-88de-4f3e-8b32-ca9f9e15a649"). InnerVolumeSpecName "kube-api-access-6lkgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:09.847663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.847609    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-podinfo" pod "4ca7984a-88de-4f3e-8b32-ca9f9e15a649" (UID: "4ca7984a-88de-4f3e-8b32-ca9f9e15a649"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:41:09.946099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.946061    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.946099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.946092    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.946349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.946107    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lkgz\" (UniqueName: \"kubernetes.io/projected/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-kube-api-access-6lkgz\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.946349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.946120    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.946349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.946129    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.946349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.946137    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:09.946349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:09.946145    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca7984a-88de-4f3e-8b32-ca9f9e15a649-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:10.244032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.243986    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:10.678494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.678469    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"
Nov 03 13:41:10.678942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.678501    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg" event={"ID":"4ca7984a-88de-4f3e-8b32-ca9f9e15a649","Type":"ContainerDied","Data":"4a00b5f17d4377a2e8e0654d3bf288cb9df90be7067ca6b367c579cbe7ced6be"}
Nov 03 13:41:10.678942 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.678558    5420 scope.go:122] "RemoveContainer" containerID="ea580a6504a2c949bfa62913b2d572d861726456f18ea43ebd1a9146d60e5b13"
Nov 03 13:41:10.715109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.715074    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"]
Nov 03 13:41:10.719272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.719245    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-7d8xg"]
Nov 03 13:41:10.838239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.835604    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:10.901963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.901936    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:10.959923 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.959879    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-xks-ca\") pod \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") "
Nov 03 13:41:10.960135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.960051    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94de26d1-fcbf-460d-aa5b-821b4647cb7a-kube-api-access-t9sll\" (UniqueName: \"kubernetes.io/projected/94de26d1-fcbf-460d-aa5b-821b4647cb7a-kube-api-access-t9sll\") pod \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") "
Nov 03 13:41:10.960254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.960168    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-openshift-ca\") pod \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\" (UID: \"94de26d1-fcbf-460d-aa5b-821b4647cb7a\") "
Nov 03 13:41:10.960695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.960667    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-openshift-ca" pod "94de26d1-fcbf-460d-aa5b-821b4647cb7a" (UID: "94de26d1-fcbf-460d-aa5b-821b4647cb7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:10.960879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.960860    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:10.963197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.963165    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-xks-ca" pod "94de26d1-fcbf-460d-aa5b-821b4647cb7a" (UID: "94de26d1-fcbf-460d-aa5b-821b4647cb7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:10.963347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:10.963326    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94de26d1-fcbf-460d-aa5b-821b4647cb7a-kube-api-access-t9sll" pod "94de26d1-fcbf-460d-aa5b-821b4647cb7a" (UID: "94de26d1-fcbf-460d-aa5b-821b4647cb7a"). InnerVolumeSpecName "kube-api-access-t9sll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:11.062609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.062563    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94de26d1-fcbf-460d-aa5b-821b4647cb7a-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:11.062609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.062603    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9sll\" (UniqueName: \"kubernetes.io/projected/94de26d1-fcbf-460d-aa5b-821b4647cb7a-kube-api-access-t9sll\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:11.085705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.085676    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" path="/var/lib/kubelet/pods/4ca7984a-88de-4f3e-8b32-ca9f9e15a649/volumes"
Nov 03 13:41:11.401093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.400986    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:11.687839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.687811    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"
Nov 03 13:41:11.688261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.687841    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb" event={"ID":"94de26d1-fcbf-460d-aa5b-821b4647cb7a","Type":"ContainerDied","Data":"f2f95a83b531fb144e4dd745dfa34f2574c0c8e42db29047aa32b9ce5fe8eaa4"}
Nov 03 13:41:11.688261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.687891    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2f95a83b531fb144e4dd745dfa34f2574c0c8e42db29047aa32b9ce5fe8eaa4"
Nov 03 13:41:11.937748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.937720    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:41:11.975790 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.975731    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-kube-api-access-4tf2k\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-kube-api-access-4tf2k\") pod \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") "
Nov 03 13:41:11.975998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.975802    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-socket\") pod \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") "
Nov 03 13:41:11.975998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.975890    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-data\") pod \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") "
Nov 03 13:41:11.979488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.976001    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-envoy\") pod \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") "
Nov 03 13:41:11.979488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.976123    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-credential-socket\") pod \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") "
Nov 03 13:41:11.979488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.976183    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-token\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-token\") pod \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") "
Nov 03 13:41:11.979488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.976320    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-certs\") pod \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") "
Nov 03 13:41:11.979488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.976382    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-podinfo\") pod \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") "
Nov 03 13:41:11.979488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.976429    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istiod-ca-cert\") pod \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\" (UID: \"0b26dd89-8856-4b20-b5be-62a4abf9c9a6\") "
Nov 03 13:41:12.000755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.998434    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-data" pod "0b26dd89-8856-4b20-b5be-62a4abf9c9a6" (UID: "0b26dd89-8856-4b20-b5be-62a4abf9c9a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:12.000755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.998973    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-socket" pod "0b26dd89-8856-4b20-b5be-62a4abf9c9a6" (UID: "0b26dd89-8856-4b20-b5be-62a4abf9c9a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:12.000755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:11.999176    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-credential-socket" pod "0b26dd89-8856-4b20-b5be-62a4abf9c9a6" (UID: "0b26dd89-8856-4b20-b5be-62a4abf9c9a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:12.000755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.000446    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-certs" pod "0b26dd89-8856-4b20-b5be-62a4abf9c9a6" (UID: "0b26dd89-8856-4b20-b5be-62a4abf9c9a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:12.008163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.008101    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istiod-ca-cert" pod "0b26dd89-8856-4b20-b5be-62a4abf9c9a6" (UID: "0b26dd89-8856-4b20-b5be-62a4abf9c9a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:12.008950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.008923    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:12.011763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.011243    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-podinfo" pod "0b26dd89-8856-4b20-b5be-62a4abf9c9a6" (UID: "0b26dd89-8856-4b20-b5be-62a4abf9c9a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:41:12.012546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.012518    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-kube-api-access-4tf2k" pod "0b26dd89-8856-4b20-b5be-62a4abf9c9a6" (UID: "0b26dd89-8856-4b20-b5be-62a4abf9c9a6"). InnerVolumeSpecName "kube-api-access-4tf2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:12.012663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.012647    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-token" pod "0b26dd89-8856-4b20-b5be-62a4abf9c9a6" (UID: "0b26dd89-8856-4b20-b5be-62a4abf9c9a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:12.012900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.012884    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-envoy" pod "0b26dd89-8856-4b20-b5be-62a4abf9c9a6" (UID: "0b26dd89-8856-4b20-b5be-62a4abf9c9a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:12.078056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.078021    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:12.078056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.078054    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tf2k\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-kube-api-access-4tf2k\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:12.078334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.078071    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:12.078334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.078084    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:12.078334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.078098    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:12.078334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.078111    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:12.078334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.078124    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:12.078334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.078137    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:12.078334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.078150    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b26dd89-8856-4b20-b5be-62a4abf9c9a6-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:12.162896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.162860    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"]
Nov 03 13:41:12.170011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.169497    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"]
Nov 03 13:41:12.192041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.191997    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"]
Nov 03 13:41:12.308621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.308593    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:12.699532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.699496    5420 generic.go:356] "Generic (PLEG): container finished" podID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerID="ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8" exitCode=0
Nov 03 13:41:12.699966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.699555    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"
Nov 03 13:41:12.699966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.699572    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" event={"ID":"0b26dd89-8856-4b20-b5be-62a4abf9c9a6","Type":"ContainerDied","Data":"ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8"}
Nov 03 13:41:12.699966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.699659    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x" event={"ID":"0b26dd89-8856-4b20-b5be-62a4abf9c9a6","Type":"ContainerDied","Data":"c1c5e758368f53a62ec0b10b22a47da02697794682ec93938d74b299431bfeb2"}
Nov 03 13:41:12.699966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.699682    5420 scope.go:122] "RemoveContainer" containerID="ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8"
Nov 03 13:41:12.713027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.713007    5420 scope.go:122] "RemoveContainer" containerID="90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98"
Nov 03 13:41:12.723304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.723283    5420 scope.go:122] "RemoveContainer" containerID="ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8"
Nov 03 13:41:12.723622 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:12.723600    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8\": container with ID starting with ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8 not found: ID does not exist" containerID="ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8"
Nov 03 13:41:12.723673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.723632    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8"} err="failed to get container status \"ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8\": rpc error: code = NotFound desc = could not find container \"ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8\": container with ID starting with ccd5011bec4872e7ba76e5274ece0780a16949d9236c0d44685fa25a73595af8 not found: ID does not exist"
Nov 03 13:41:12.723673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.723653    5420 scope.go:122] "RemoveContainer" containerID="90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98"
Nov 03 13:41:12.723908 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:12.723884    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98\": container with ID starting with 90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98 not found: ID does not exist" containerID="90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98"
Nov 03 13:41:12.724047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.723916    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98"} err="failed to get container status \"90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98\": rpc error: code = NotFound desc = could not find container \"90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98\": container with ID starting with 90fd6cdbe7644ffdff6005b5f9f5034f22baef02cf42139e2e8e0c2cdb108d98 not found: ID does not exist"
Nov 03 13:41:12.727229 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.727176    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"]
Nov 03 13:41:12.732574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.732552    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-cf53a4-openshift-default-866d74b569-rwx7x"]
Nov 03 13:41:12.815925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.815891    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:12.949810 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:12.949721    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:13.085101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:13.085071    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" path="/var/lib/kubelet/pods/0b26dd89-8856-4b20-b5be-62a4abf9c9a6/volumes"
Nov 03 13:41:13.408544 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:13.405678    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:14.047762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:14.047736    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:14.690389 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:14.690360    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:15.391611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:15.391577    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:16.122721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.111712    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:16.164853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.164815    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"]
Nov 03 13:41:16.165632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165614    5420 cpu_manager.go:414] "containerMap: removing container" podUID="a29f33dc-fd1f-40fc-8779-034d313df505" containerName="revoke-keys"
Nov 03 13:41:16.165700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165633    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29f33dc-fd1f-40fc-8779-034d313df505" containerName="revoke-keys"
Nov 03 13:41:16.165700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165646    5420 cpu_manager.go:414] "containerMap: removing container" podUID="94de26d1-fcbf-460d-aa5b-821b4647cb7a" containerName="revoke-keys"
Nov 03 13:41:16.165700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165651    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94de26d1-fcbf-460d-aa5b-821b4647cb7a" containerName="revoke-keys"
Nov 03 13:41:16.165700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165663    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy"
Nov 03 13:41:16.165700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165669    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy"
Nov 03 13:41:16.165700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165679    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy"
Nov 03 13:41:16.165700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165684    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy"
Nov 03 13:41:16.165700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165703    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d" containerName="revoke-keys"
Nov 03 13:41:16.166050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165708    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d" containerName="revoke-keys"
Nov 03 13:41:16.166050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165720    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy"
Nov 03 13:41:16.166050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165724    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy"
Nov 03 13:41:16.166050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165819    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29f33dc-fd1f-40fc-8779-034d313df505" containerName="revoke-keys"
Nov 03 13:41:16.166050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165827    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="94de26d1-fcbf-460d-aa5b-821b4647cb7a" containerName="revoke-keys"
Nov 03 13:41:16.166050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165834    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy"
Nov 03 13:41:16.166050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165842    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca7984a-88de-4f3e-8b32-ca9f9e15a649" containerName="istio-proxy"
Nov 03 13:41:16.166050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.165850    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d" containerName="revoke-keys"
Nov 03 13:41:16.171094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.171069    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.174776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.174748    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"]
Nov 03 13:41:16.175455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.175436    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:41:16.176707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.176690    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:41:16.176833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.176762    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:41:16.176915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.176873    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:41:16.243338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.243308    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2bb6759-4572h\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.243515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.243376    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2bb6759-4572h\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.243599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.243511    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhp5f\" (UniqueName: \"kubernetes.io/projected/d90ea2f2-e22f-4ff6-9212-0a1c69998203-kube-api-access-qhp5f\") pod \"maas-api-revoke-sub-c2bb6759-4572h\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.345515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.345475    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2bb6759-4572h\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.345684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.345585    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2bb6759-4572h\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.345684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.345632    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhp5f\" (UniqueName: \"kubernetes.io/projected/d90ea2f2-e22f-4ff6-9212-0a1c69998203-kube-api-access-qhp5f\") pod \"maas-api-revoke-sub-c2bb6759-4572h\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.346177 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.346154    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2bb6759-4572h\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.347934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.347911    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2bb6759-4572h\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.356362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.356340    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhp5f\" (UniqueName: \"kubernetes.io/projected/d90ea2f2-e22f-4ff6-9212-0a1c69998203-kube-api-access-qhp5f\") pod \"maas-api-revoke-sub-c2bb6759-4572h\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.495837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.495800    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:16.785528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.782272    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:16.801661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:16.801616    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"]
Nov 03 13:41:17.415288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:17.411772    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:17.742455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:17.742420    5420 generic.go:356] "Generic (PLEG): container finished" podID="d90ea2f2-e22f-4ff6-9212-0a1c69998203" containerID="ea7bb5a9c3693e3c8cc12264b96572581f5051ea76154878cb3952e52e3458ec" exitCode=0
Nov 03 13:41:17.742883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:17.742465    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h" event={"ID":"d90ea2f2-e22f-4ff6-9212-0a1c69998203","Type":"ContainerDied","Data":"ea7bb5a9c3693e3c8cc12264b96572581f5051ea76154878cb3952e52e3458ec"}
Nov 03 13:41:17.742883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:17.742539    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h" event={"ID":"d90ea2f2-e22f-4ff6-9212-0a1c69998203","Type":"ContainerStarted","Data":"d77817ba2817e39027d9f8fe2f0b5db2f3b85824ba264e80d314ddaa94b8ad74"}
Nov 03 13:41:17.989089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:17.989047    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:18.934793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:18.934728    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:18.977693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:18.977658    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"]
Nov 03 13:41:18.978771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:18.978681    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d90ea2f2-e22f-4ff6-9212-0a1c69998203" containerName="revoke-keys"
Nov 03 13:41:18.978955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:18.978937    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90ea2f2-e22f-4ff6-9212-0a1c69998203" containerName="revoke-keys"
Nov 03 13:41:18.979320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:18.979304    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b26dd89-8856-4b20-b5be-62a4abf9c9a6" containerName="istio-proxy"
Nov 03 13:41:18.979439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:18.979428    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90ea2f2-e22f-4ff6-9212-0a1c69998203" containerName="revoke-keys"
Nov 03 13:41:18.983867 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:18.983847    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"]
Nov 03 13:41:18.984116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:18.984092    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:18.985064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:18.985042    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:19.040531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.040450    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"]
Nov 03 13:41:19.051484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.051448    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"]
Nov 03 13:41:19.086435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.086407    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-openshift-ca\") pod \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") "
Nov 03 13:41:19.089050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.089026    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-xks-ca\") pod \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") "
Nov 03 13:41:19.089657 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.088874    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-openshift-ca" pod "d90ea2f2-e22f-4ff6-9212-0a1c69998203" (UID: "d90ea2f2-e22f-4ff6-9212-0a1c69998203"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:19.090225 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.090191    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90ea2f2-e22f-4ff6-9212-0a1c69998203-kube-api-access-qhp5f\" (UniqueName: \"kubernetes.io/projected/d90ea2f2-e22f-4ff6-9212-0a1c69998203-kube-api-access-qhp5f\") pod \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\" (UID: \"d90ea2f2-e22f-4ff6-9212-0a1c69998203\") "
Nov 03 13:41:19.091460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.091435    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e75044-127e-4acf-98be-f928956d6396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bab5a758-kgjdv\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.091756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.091736    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e75044-127e-4acf-98be-f928956d6396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bab5a758-kgjdv\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.092254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.092194    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5p5q\" (UniqueName: \"kubernetes.io/projected/06e75044-127e-4acf-98be-f928956d6396-kube-api-access-w5p5q\") pod \"maas-api-revoke-sub-bab5a758-kgjdv\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.092379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.092364    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:19.102651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.102621    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-xks-ca" pod "d90ea2f2-e22f-4ff6-9212-0a1c69998203" (UID: "d90ea2f2-e22f-4ff6-9212-0a1c69998203"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:19.103530 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.103498    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90ea2f2-e22f-4ff6-9212-0a1c69998203-kube-api-access-qhp5f" pod "d90ea2f2-e22f-4ff6-9212-0a1c69998203" (UID: "d90ea2f2-e22f-4ff6-9212-0a1c69998203"). InnerVolumeSpecName "kube-api-access-qhp5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:19.171227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.171176    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:19.193149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.193119    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e75044-127e-4acf-98be-f928956d6396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bab5a758-kgjdv\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.193398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.193223    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e75044-127e-4acf-98be-f928956d6396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bab5a758-kgjdv\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.193398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.193292    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5p5q\" (UniqueName: \"kubernetes.io/projected/06e75044-127e-4acf-98be-f928956d6396-kube-api-access-w5p5q\") pod \"maas-api-revoke-sub-bab5a758-kgjdv\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.193695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.193669    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90ea2f2-e22f-4ff6-9212-0a1c69998203-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:19.193775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.193700    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhp5f\" (UniqueName: \"kubernetes.io/projected/d90ea2f2-e22f-4ff6-9212-0a1c69998203-kube-api-access-qhp5f\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:19.193952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.193933    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e75044-127e-4acf-98be-f928956d6396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bab5a758-kgjdv\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.195818 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.195760    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e75044-127e-4acf-98be-f928956d6396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bab5a758-kgjdv\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.205518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.205496    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5p5q\" (UniqueName: \"kubernetes.io/projected/06e75044-127e-4acf-98be-f928956d6396-kube-api-access-w5p5q\") pod \"maas-api-revoke-sub-bab5a758-kgjdv\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.301561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.301489    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:19.318160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.318126    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:19.474596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.474544    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:19.715676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.715649    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:19.730050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.730013    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"]
Nov 03 13:41:19.730465 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:41:19.730443    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e75044_127e_4acf_98be_f928956d6396.slice/crio-6180d17b6b36d3af9dc0a2d8019db7104901379e7cd90d0b122290404596e960 WatchSource:0}: Error finding container 6180d17b6b36d3af9dc0a2d8019db7104901379e7cd90d0b122290404596e960: Status 404 returned error can't find the container with id 6180d17b6b36d3af9dc0a2d8019db7104901379e7cd90d0b122290404596e960
Nov 03 13:41:19.758959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.758933    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv" event={"ID":"06e75044-127e-4acf-98be-f928956d6396","Type":"ContainerStarted","Data":"6180d17b6b36d3af9dc0a2d8019db7104901379e7cd90d0b122290404596e960"}
Nov 03 13:41:19.760422 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.760403    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"
Nov 03 13:41:19.760564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.760415    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h" event={"ID":"d90ea2f2-e22f-4ff6-9212-0a1c69998203","Type":"ContainerDied","Data":"d77817ba2817e39027d9f8fe2f0b5db2f3b85824ba264e80d314ddaa94b8ad74"}
Nov 03 13:41:19.760564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.760462    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d77817ba2817e39027d9f8fe2f0b5db2f3b85824ba264e80d314ddaa94b8ad74"
Nov 03 13:41:19.896357 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:19.892257    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:20.596605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:20.596493    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:20.769847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:20.769812    5420 generic.go:356] "Generic (PLEG): container finished" podID="06e75044-127e-4acf-98be-f928956d6396" containerID="cae3154cf953fd324cdb4edf2e0f116247469173f1c0ccaecf00e68dedf7390d" exitCode=0
Nov 03 13:41:20.770023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:20.769902    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv" event={"ID":"06e75044-127e-4acf-98be-f928956d6396","Type":"ContainerDied","Data":"cae3154cf953fd324cdb4edf2e0f116247469173f1c0ccaecf00e68dedf7390d"}
Nov 03 13:41:21.200269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:21.200238    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:21.799265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:21.794009    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:21.974028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:21.973991    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:22.141414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.141341    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e75044-127e-4acf-98be-f928956d6396-kube-api-access-w5p5q\" (UniqueName: \"kubernetes.io/projected/06e75044-127e-4acf-98be-f928956d6396-kube-api-access-w5p5q\") pod \"06e75044-127e-4acf-98be-f928956d6396\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") "
Nov 03 13:41:22.141572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.141466    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e75044-127e-4acf-98be-f928956d6396-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e75044-127e-4acf-98be-f928956d6396-maas-api-xks-ca\") pod \"06e75044-127e-4acf-98be-f928956d6396\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") "
Nov 03 13:41:22.141572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.141522    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e75044-127e-4acf-98be-f928956d6396-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e75044-127e-4acf-98be-f928956d6396-maas-api-openshift-ca\") pod \"06e75044-127e-4acf-98be-f928956d6396\" (UID: \"06e75044-127e-4acf-98be-f928956d6396\") "
Nov 03 13:41:22.141908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.141882    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e75044-127e-4acf-98be-f928956d6396-maas-api-openshift-ca" pod "06e75044-127e-4acf-98be-f928956d6396" (UID: "06e75044-127e-4acf-98be-f928956d6396"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:22.143513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.143489    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e75044-127e-4acf-98be-f928956d6396-maas-api-xks-ca" pod "06e75044-127e-4acf-98be-f928956d6396" (UID: "06e75044-127e-4acf-98be-f928956d6396"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:22.143597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.143508    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e75044-127e-4acf-98be-f928956d6396-kube-api-access-w5p5q" pod "06e75044-127e-4acf-98be-f928956d6396" (UID: "06e75044-127e-4acf-98be-f928956d6396"). InnerVolumeSpecName "kube-api-access-w5p5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:22.243576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.243546    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5p5q\" (UniqueName: \"kubernetes.io/projected/06e75044-127e-4acf-98be-f928956d6396-kube-api-access-w5p5q\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:22.243576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.243574    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e75044-127e-4acf-98be-f928956d6396-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:22.243781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.243585    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e75044-127e-4acf-98be-f928956d6396-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:22.429857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.429763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:22.442549 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.442518    5420 scope.go:122] "RemoveContainer" containerID="4ac8bfadd66a80809f03fc3da4603a3037e0e5bf3630792f6c8093e7a811653e"
Nov 03 13:41:22.471628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.471569    5420 scope.go:122] "RemoveContainer" containerID="8fb5212fb05732ad42ce22b6e464d3f542b1f64f19868dd27f57e8276042a91d"
Nov 03 13:41:22.482025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.481995    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:41:22.493714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.493687    5420 scope.go:122] "RemoveContainer" containerID="069e2b3db24ccf688410195e65a328f856c5d625b6a5a2dc1420e871c1c6d235"
Nov 03 13:41:22.506532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.506490    5420 scope.go:122] "RemoveContainer" containerID="84b36ea7cd890535538783c620abdc84682537fee2bec309072e6f25cef62513"
Nov 03 13:41:22.516821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.516802    5420 scope.go:122] "RemoveContainer" containerID="36a461aa398db81c24175f76fa560f8a6caee6d7746503f1568201d4db40a204"
Nov 03 13:41:22.526515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.526489    5420 scope.go:122] "RemoveContainer" containerID="3143d6200e37d6544cabdf2fc5afbe2500cc64d1980920be93f32e700fefd76f"
Nov 03 13:41:22.536197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.536178    5420 scope.go:122] "RemoveContainer" containerID="d1d6e3ce9067e3b5219359ee0ac90b7acab6d9023252419997e93d321cac3da2"
Nov 03 13:41:22.787079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.787054    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"
Nov 03 13:41:22.787332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.787076    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv" event={"ID":"06e75044-127e-4acf-98be-f928956d6396","Type":"ContainerDied","Data":"6180d17b6b36d3af9dc0a2d8019db7104901379e7cd90d0b122290404596e960"}
Nov 03 13:41:22.787332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:22.787126    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6180d17b6b36d3af9dc0a2d8019db7104901379e7cd90d0b122290404596e960"
Nov 03 13:41:23.029235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:23.025912    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:23.899201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:23.899157    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:24.049237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:24.041304    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4" podUID="c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" containerName="payload-processing" containerID="cri-o://41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee" gracePeriod=25
Nov 03 13:41:24.059449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:24.059406    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs" podUID="af2bd68c-986a-4fdc-aafd-d398b8b7d87e" containerName="payload-pre-processing" containerID="cri-o://ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821" gracePeriod=25
Nov 03 13:41:24.659840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:24.651766    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:24.742809 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:24.742761    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:41:25.315973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:25.314448    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:26.026762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.026712    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:26.545783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.545753    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"]
Nov 03 13:41:26.546447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.546430    5420 cpu_manager.go:414] "containerMap: removing container" podUID="06e75044-127e-4acf-98be-f928956d6396" containerName="revoke-keys"
Nov 03 13:41:26.546447 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.546447    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e75044-127e-4acf-98be-f928956d6396" containerName="revoke-keys"
Nov 03 13:41:26.546581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.546572    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e75044-127e-4acf-98be-f928956d6396" containerName="revoke-keys"
Nov 03 13:41:26.552157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.552139    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.557829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.557807    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:41:26.557969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.557861    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:41:26.558042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.557971    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:41:26.558042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.558014    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:41:26.560699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.560674    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"]
Nov 03 13:41:26.634870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.634838    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a16df6-6862-43c5-b438-9a6741980520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca8c2fb-bhqt4\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.635049 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.635032    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpgtk\" (UniqueName: \"kubernetes.io/projected/08a16df6-6862-43c5-b438-9a6741980520-kube-api-access-kpgtk\") pod \"maas-api-revoke-sub-fca8c2fb-bhqt4\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.635114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.635100    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a16df6-6862-43c5-b438-9a6741980520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca8c2fb-bhqt4\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.736248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.736187    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpgtk\" (UniqueName: \"kubernetes.io/projected/08a16df6-6862-43c5-b438-9a6741980520-kube-api-access-kpgtk\") pod \"maas-api-revoke-sub-fca8c2fb-bhqt4\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.736439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.736323    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a16df6-6862-43c5-b438-9a6741980520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca8c2fb-bhqt4\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.736439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.736411    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a16df6-6862-43c5-b438-9a6741980520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca8c2fb-bhqt4\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.737013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.736989    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a16df6-6862-43c5-b438-9a6741980520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca8c2fb-bhqt4\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.738870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.738847    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a16df6-6862-43c5-b438-9a6741980520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca8c2fb-bhqt4\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.744104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.744081    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpgtk\" (UniqueName: \"kubernetes.io/projected/08a16df6-6862-43c5-b438-9a6741980520-kube-api-access-kpgtk\") pod \"maas-api-revoke-sub-fca8c2fb-bhqt4\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.823116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.823033    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:26.875443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.875414    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:26.970487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:26.970453    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:27.161602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.161577    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"]
Nov 03 13:41:27.163355 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:41:27.163329    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a16df6_6862_43c5_b438_9a6741980520.slice/crio-0901ffbd54693789aa3e452ea0fcff01a7ae4e70e7564554699b22bd1cc12a90 WatchSource:0}: Error finding container 0901ffbd54693789aa3e452ea0fcff01a7ae4e70e7564554699b22bd1cc12a90: Status 404 returned error can't find the container with id 0901ffbd54693789aa3e452ea0fcff01a7ae4e70e7564554699b22bd1cc12a90
Nov 03 13:41:27.184799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.184757    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv" podUID="a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" containerName="main" containerID="cri-o://41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79" gracePeriod=45
Nov 03 13:41:27.192249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.192200    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w" podUID="9a0fafd5-a802-41e7-b2b2-02a11ebf6217" containerName="main" containerID="cri-o://7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916" gracePeriod=45
Nov 03 13:41:27.213393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.213360    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt" podUID="8506a60e-792e-4877-804a-d998bda6b1aa" containerName="main" containerID="cri-o://81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c" gracePeriod=45
Nov 03 13:41:27.383241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.379372    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:27.567510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.567314    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:41:27.572484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.572455    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:41:27.576666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.576637    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:41:27.652826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.652786    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8506a60e-792e-4877-804a-d998bda6b1aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/8506a60e-792e-4877-804a-d998bda6b1aa-tls-certs\") pod \"8506a60e-792e-4877-804a-d998bda6b1aa\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") "
Nov 03 13:41:27.653002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.652891    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tmp-dir\") pod \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") "
Nov 03 13:41:27.653002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.652966    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-model-cache\") pod \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") "
Nov 03 13:41:27.653102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.653022    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-dshm\") pod \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") "
Nov 03 13:41:27.653102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.653072    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tls-certs\") pod \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") "
Nov 03 13:41:27.653233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.653120    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-dshm\") pod \"8506a60e-792e-4877-804a-d998bda6b1aa\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") "
Nov 03 13:41:27.653233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.653158    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-home\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-home\") pod \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") "
Nov 03 13:41:27.653233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.653198    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-dshm\") pod \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") "
Nov 03 13:41:27.653402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.653325    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-home\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-home\") pod \"8506a60e-792e-4877-804a-d998bda6b1aa\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") "
Nov 03 13:41:27.653402 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.653357    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-model-cache" pod "9a0fafd5-a802-41e7-b2b2-02a11ebf6217" (UID: "9a0fafd5-a802-41e7-b2b2-02a11ebf6217"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.653630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.653598    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tmp-dir" pod "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" (UID: "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.653825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.653804    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-home" pod "9a0fafd5-a802-41e7-b2b2-02a11ebf6217" (UID: "9a0fafd5-a802-41e7-b2b2-02a11ebf6217"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.655312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.655280    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-model-cache\") pod \"8506a60e-792e-4877-804a-d998bda6b1aa\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") "
Nov 03 13:41:27.655439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.655343    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8506a60e-792e-4877-804a-d998bda6b1aa-kube-api-access-6mdvk\" (UniqueName: \"kubernetes.io/projected/8506a60e-792e-4877-804a-d998bda6b1aa-kube-api-access-6mdvk\") pod \"8506a60e-792e-4877-804a-d998bda6b1aa\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") "
Nov 03 13:41:27.655439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.655414    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tls-certs\") pod \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") "
Nov 03 13:41:27.655539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.655501    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-tmp-dir\") pod \"8506a60e-792e-4877-804a-d998bda6b1aa\" (UID: \"8506a60e-792e-4877-804a-d998bda6b1aa\") "
Nov 03 13:41:27.655539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.655530    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-home\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-home\") pod \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") "
Nov 03 13:41:27.655710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.655569    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-kube-api-access-b7s59\" (UniqueName: \"kubernetes.io/projected/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-kube-api-access-b7s59\") pod \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") "
Nov 03 13:41:27.655710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.655614    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-kube-api-access-s2wbv\" (UniqueName: \"kubernetes.io/projected/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-kube-api-access-s2wbv\") pod \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") "
Nov 03 13:41:27.655710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.655673    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-model-cache\") pod \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\" (UID: \"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6\") "
Nov 03 13:41:27.655865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.655742    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tmp-dir\") pod \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\" (UID: \"9a0fafd5-a802-41e7-b2b2-02a11ebf6217\") "
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656041    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-dshm" pod "8506a60e-792e-4877-804a-d998bda6b1aa" (UID: "8506a60e-792e-4877-804a-d998bda6b1aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656359    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-model-cache" pod "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" (UID: "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656483    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tmp-dir" pod "9a0fafd5-a802-41e7-b2b2-02a11ebf6217" (UID: "9a0fafd5-a802-41e7-b2b2-02a11ebf6217"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656626    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-home" pod "8506a60e-792e-4877-804a-d998bda6b1aa" (UID: "8506a60e-792e-4877-804a-d998bda6b1aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656640    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656663    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656676    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656691    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656704    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656717    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.656927    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-tmp-dir" pod "8506a60e-792e-4877-804a-d998bda6b1aa" (UID: "8506a60e-792e-4877-804a-d998bda6b1aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.657382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.657104    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-model-cache" pod "8506a60e-792e-4877-804a-d998bda6b1aa" (UID: "8506a60e-792e-4877-804a-d998bda6b1aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.659971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.658015    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-home" pod "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" (UID: "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.659971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.658290    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-dshm" pod "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" (UID: "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.659971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.658786    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-dshm" pod "9a0fafd5-a802-41e7-b2b2-02a11ebf6217" (UID: "9a0fafd5-a802-41e7-b2b2-02a11ebf6217"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:27.660623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.660603    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8506a60e-792e-4877-804a-d998bda6b1aa-tls-certs" pod "8506a60e-792e-4877-804a-d998bda6b1aa" (UID: "8506a60e-792e-4877-804a-d998bda6b1aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:27.660725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.660628    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tls-certs" pod "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" (UID: "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:27.660875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.660845    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tls-certs" pod "9a0fafd5-a802-41e7-b2b2-02a11ebf6217" (UID: "9a0fafd5-a802-41e7-b2b2-02a11ebf6217"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:27.660961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.660937    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-kube-api-access-s2wbv" pod "9a0fafd5-a802-41e7-b2b2-02a11ebf6217" (UID: "9a0fafd5-a802-41e7-b2b2-02a11ebf6217"). InnerVolumeSpecName "kube-api-access-s2wbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:27.662182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.662156    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-kube-api-access-b7s59" pod "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" (UID: "a5fa55b7-f94d-495d-b8ca-8c7b97632cb6"). InnerVolumeSpecName "kube-api-access-b7s59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:27.662774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.662717    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8506a60e-792e-4877-804a-d998bda6b1aa-kube-api-access-6mdvk" pod "8506a60e-792e-4877-804a-d998bda6b1aa" (UID: "8506a60e-792e-4877-804a-d998bda6b1aa"). InnerVolumeSpecName "kube-api-access-6mdvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.758995    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759032    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759047    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759062    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759075    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759094    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mdvk\" (UniqueName: \"kubernetes.io/projected/8506a60e-792e-4877-804a-d998bda6b1aa-kube-api-access-6mdvk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759107    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759120    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8506a60e-792e-4877-804a-d998bda6b1aa-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759133    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759147    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7s59\" (UniqueName: \"kubernetes.io/projected/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6-kube-api-access-b7s59\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759160    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wbv\" (UniqueName: \"kubernetes.io/projected/9a0fafd5-a802-41e7-b2b2-02a11ebf6217-kube-api-access-s2wbv\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.761395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.759173    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8506a60e-792e-4877-804a-d998bda6b1aa-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:27.849500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.849460    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4" event={"ID":"08a16df6-6862-43c5-b438-9a6741980520","Type":"ContainerStarted","Data":"a1b92f2818f16027353549364caa202fac6dd3158d6e46699f7bdd5dc62fd35b"}
Nov 03 13:41:27.849699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.849525    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4" event={"ID":"08a16df6-6862-43c5-b438-9a6741980520","Type":"ContainerStarted","Data":"0901ffbd54693789aa3e452ea0fcff01a7ae4e70e7564554699b22bd1cc12a90"}
Nov 03 13:41:27.850961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.850933    5420 generic.go:356] "Generic (PLEG): container finished" podID="8506a60e-792e-4877-804a-d998bda6b1aa" containerID="81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c" exitCode=0
Nov 03 13:41:27.851083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.850975    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"
Nov 03 13:41:27.851083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.851018    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt" event={"ID":"8506a60e-792e-4877-804a-d998bda6b1aa","Type":"ContainerDied","Data":"81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c"}
Nov 03 13:41:27.851083 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.851062    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt" event={"ID":"8506a60e-792e-4877-804a-d998bda6b1aa","Type":"ContainerDied","Data":"1b563e62008dc214bba97e229060d31787af95adea117d8b7d6e5cee4343bc7c"}
Nov 03 13:41:27.851279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.851083    5420 scope.go:122] "RemoveContainer" containerID="81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c"
Nov 03 13:41:27.852681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.852656    5420 generic.go:356] "Generic (PLEG): container finished" podID="9a0fafd5-a802-41e7-b2b2-02a11ebf6217" containerID="7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916" exitCode=0
Nov 03 13:41:27.852816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.852710    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"
Nov 03 13:41:27.852958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.852763    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w" event={"ID":"9a0fafd5-a802-41e7-b2b2-02a11ebf6217","Type":"ContainerDied","Data":"7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916"}
Nov 03 13:41:27.853054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.852971    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w" event={"ID":"9a0fafd5-a802-41e7-b2b2-02a11ebf6217","Type":"ContainerDied","Data":"6cdfc892af9a76121f926f7a24359a172c5f4cdb422583a40c3a00d47079b549"}
Nov 03 13:41:27.859376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.859350    5420 generic.go:356] "Generic (PLEG): container finished" podID="a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" containerID="41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79" exitCode=0
Nov 03 13:41:27.859513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.859451    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv" event={"ID":"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6","Type":"ContainerDied","Data":"41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79"}
Nov 03 13:41:27.859513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.859478    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv" event={"ID":"a5fa55b7-f94d-495d-b8ca-8c7b97632cb6","Type":"ContainerDied","Data":"7a8d135c3230e341cd17147bd9a8bdb5518abf6444bf1e4748948e9a30eab859"}
Nov 03 13:41:27.859513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.859490    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"
Nov 03 13:41:27.864718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.864699    5420 scope.go:122] "RemoveContainer" containerID="81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c"
Nov 03 13:41:27.865391 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:27.865029    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c\": container with ID starting with 81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c not found: ID does not exist" containerID="81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c"
Nov 03 13:41:27.865391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.865314    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c"} err="failed to get container status \"81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c\": rpc error: code = NotFound desc = could not find container \"81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c\": container with ID starting with 81714c65c77d4e1c2e75893c5d1f40ab1b6898b33ae4721cb68c6df878d7494c not found: ID does not exist"
Nov 03 13:41:27.865391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.865339    5420 scope.go:122] "RemoveContainer" containerID="7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916"
Nov 03 13:41:27.873581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.873534    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4" podStartSLOduration=1.873521065 podStartE2EDuration="1.873521065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:27.866637982 +0000 UTC m=+1928.944403004" watchObservedRunningTime="2026-11-03 13:41:27.873521065 +0000 UTC m=+1928.951286090"
Nov 03 13:41:27.881820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.881699    5420 scope.go:122] "RemoveContainer" containerID="7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916"
Nov 03 13:41:27.882025 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:27.882005    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916\": container with ID starting with 7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916 not found: ID does not exist" containerID="7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916"
Nov 03 13:41:27.882097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.882032    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916"} err="failed to get container status \"7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916\": rpc error: code = NotFound desc = could not find container \"7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916\": container with ID starting with 7e1ce64b3eb898697a8640d962297219ae1479899cd8ee88f851c6ada5f65916 not found: ID does not exist"
Nov 03 13:41:27.882097 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.882048    5420 scope.go:122] "RemoveContainer" containerID="41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79"
Nov 03 13:41:27.885157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.885134    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"]
Nov 03 13:41:27.890412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.890387    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/simulator-w3-d3ca48-kserve-6f6c568d69-wqwpt"]
Nov 03 13:41:27.893619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.893602    5420 scope.go:122] "RemoveContainer" containerID="41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79"
Nov 03 13:41:27.893916 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:27.893894    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79\": container with ID starting with 41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79 not found: ID does not exist" containerID="41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79"
Nov 03 13:41:27.893965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.893926    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79"} err="failed to get container status \"41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79\": rpc error: code = NotFound desc = could not find container \"41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79\": container with ID starting with 41c2c9871f17f7c9203e7108f07c69690493b9f2529b8fd3292b0a9b0e020e79 not found: ID does not exist"
Nov 03 13:41:27.901920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.901898    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"]
Nov 03 13:41:27.905915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.905892    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-w3-d3ca48-kserve-68dfd77cc5-w5b8w"]
Nov 03 13:41:27.920336 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.920260    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"]
Nov 03 13:41:27.924982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.924956    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/distinct-2-w3-d3ca48-kserve-64549d6784-8qtjv"]
Nov 03 13:41:27.995486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:27.994631    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:28.317270 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:28.317232    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4" podUID="c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:41:28.318839 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:28.318813    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs" podUID="af2bd68c-986a-4fdc-aafd-d398b8b7d87e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:41:28.628507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:28.628403    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:28.873010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:28.872973    5420 generic.go:356] "Generic (PLEG): container finished" podID="08a16df6-6862-43c5-b438-9a6741980520" containerID="a1b92f2818f16027353549364caa202fac6dd3158d6e46699f7bdd5dc62fd35b" exitCode=0
Nov 03 13:41:28.873296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:28.873094    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4" event={"ID":"08a16df6-6862-43c5-b438-9a6741980520","Type":"ContainerDied","Data":"a1b92f2818f16027353549364caa202fac6dd3158d6e46699f7bdd5dc62fd35b"}
Nov 03 13:41:29.087494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:29.087458    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8506a60e-792e-4877-804a-d998bda6b1aa" path="/var/lib/kubelet/pods/8506a60e-792e-4877-804a-d998bda6b1aa/volumes"
Nov 03 13:41:29.087949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:29.087934    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0fafd5-a802-41e7-b2b2-02a11ebf6217" path="/var/lib/kubelet/pods/9a0fafd5-a802-41e7-b2b2-02a11ebf6217/volumes"
Nov 03 13:41:29.088379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:29.088365    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" path="/var/lib/kubelet/pods/a5fa55b7-f94d-495d-b8ca-8c7b97632cb6/volumes"
Nov 03 13:41:29.222242 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:29.220829    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:29.815869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:29.815833    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:30.040943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.040916    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:30.091895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.091820    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a16df6-6862-43c5-b438-9a6741980520-kube-api-access-kpgtk\" (UniqueName: \"kubernetes.io/projected/08a16df6-6862-43c5-b438-9a6741980520-kube-api-access-kpgtk\") pod \"08a16df6-6862-43c5-b438-9a6741980520\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") "
Nov 03 13:41:30.091895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.091875    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a16df6-6862-43c5-b438-9a6741980520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a16df6-6862-43c5-b438-9a6741980520-maas-api-openshift-ca\") pod \"08a16df6-6862-43c5-b438-9a6741980520\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") "
Nov 03 13:41:30.092136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.092002    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a16df6-6862-43c5-b438-9a6741980520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a16df6-6862-43c5-b438-9a6741980520-maas-api-xks-ca\") pod \"08a16df6-6862-43c5-b438-9a6741980520\" (UID: \"08a16df6-6862-43c5-b438-9a6741980520\") "
Nov 03 13:41:30.092304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.092278    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a16df6-6862-43c5-b438-9a6741980520-maas-api-openshift-ca" pod "08a16df6-6862-43c5-b438-9a6741980520" (UID: "08a16df6-6862-43c5-b438-9a6741980520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:30.092688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.092666    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a16df6-6862-43c5-b438-9a6741980520-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:30.094085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.094063    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a16df6-6862-43c5-b438-9a6741980520-maas-api-xks-ca" pod "08a16df6-6862-43c5-b438-9a6741980520" (UID: "08a16df6-6862-43c5-b438-9a6741980520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:41:30.094509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.094481    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a16df6-6862-43c5-b438-9a6741980520-kube-api-access-kpgtk" pod "08a16df6-6862-43c5-b438-9a6741980520" (UID: "08a16df6-6862-43c5-b438-9a6741980520"). InnerVolumeSpecName "kube-api-access-kpgtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:30.193757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.193726    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a16df6-6862-43c5-b438-9a6741980520-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:30.193757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.193760    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpgtk\" (UniqueName: \"kubernetes.io/projected/08a16df6-6862-43c5-b438-9a6741980520-kube-api-access-kpgtk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:30.618266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.616286    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:30.891769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.891682    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"
Nov 03 13:41:30.891769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.891712    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4" event={"ID":"08a16df6-6862-43c5-b438-9a6741980520","Type":"ContainerDied","Data":"0901ffbd54693789aa3e452ea0fcff01a7ae4e70e7564554699b22bd1cc12a90"}
Nov 03 13:41:30.891769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:30.891757    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0901ffbd54693789aa3e452ea0fcff01a7ae4e70e7564554699b22bd1cc12a90"
Nov 03 13:41:31.289858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:31.289820    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:31.916655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:31.916609    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:32.532770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:32.532733    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:33.084238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:33.082982    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:33.229615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:33.229584    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:33.696183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:33.696101    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:34.389985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:34.389880    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:35.100201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:35.100169    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:35.530814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:35.530763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:36.147860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:36.147803    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:36.747989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:36.747949    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:37.279734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:37.279698    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:37.523074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:37.523039    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:38.272267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.271594    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:38.318650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.318613    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs" podUID="af2bd68c-986a-4fdc-aafd-d398b8b7d87e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:41:38.319104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.319066    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4" podUID="c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:41:38.457275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.457231    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"]
Nov 03 13:41:38.457956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.457941    5420 cpu_manager.go:414] "containerMap: removing container" podUID="9a0fafd5-a802-41e7-b2b2-02a11ebf6217" containerName="main"
Nov 03 13:41:38.458012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.457957    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0fafd5-a802-41e7-b2b2-02a11ebf6217" containerName="main"
Nov 03 13:41:38.458012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.457968    5420 cpu_manager.go:414] "containerMap: removing container" podUID="a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" containerName="main"
Nov 03 13:41:38.458012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.457976    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" containerName="main"
Nov 03 13:41:38.458012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.457996    5420 cpu_manager.go:414] "containerMap: removing container" podUID="08a16df6-6862-43c5-b438-9a6741980520" containerName="revoke-keys"
Nov 03 13:41:38.458012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.458002    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a16df6-6862-43c5-b438-9a6741980520" containerName="revoke-keys"
Nov 03 13:41:38.458171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.458042    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8506a60e-792e-4877-804a-d998bda6b1aa" containerName="main"
Nov 03 13:41:38.458171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.458049    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8506a60e-792e-4877-804a-d998bda6b1aa" containerName="main"
Nov 03 13:41:38.458171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.458142    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a16df6-6862-43c5-b438-9a6741980520" containerName="revoke-keys"
Nov 03 13:41:38.458171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.458158    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8506a60e-792e-4877-804a-d998bda6b1aa" containerName="main"
Nov 03 13:41:38.458171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.458165    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0fafd5-a802-41e7-b2b2-02a11ebf6217" containerName="main"
Nov 03 13:41:38.458171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.458171    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fa55b7-f94d-495d-b8ca-8c7b97632cb6" containerName="main"
Nov 03 13:41:38.462040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.462013    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.496718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.496685    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"]
Nov 03 13:41:38.507322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.507289    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"]
Nov 03 13:41:38.512606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.512581    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"]
Nov 03 13:41:38.512759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.512677    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.529885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.529857    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.530019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.529914    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.530077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.530051    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.530130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.530099    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.530262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.530244    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.530343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.530297    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqkw\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-kube-api-access-fcqkw\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.530406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.530374    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.530527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.530512    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.530592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.530571    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.632392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632353    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.632392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632394    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.632602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632419    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.632602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632511    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.632602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632557    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.632755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632605    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.632755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632683    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqkw\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-kube-api-access-fcqkw\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.632755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632717    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.632755 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632741    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfls\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-kube-api-access-2hfls\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.632992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632871    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.632992 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.632951    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.633102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633017    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.633168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633124    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.633303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633280    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.633395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633333    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.633395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633341    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.633395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633362    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.633543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633455    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.633543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633478    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.633543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633518    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.633651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633555    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.633651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633640    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.633782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.633766    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.634966 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.634938    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.635266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.635251    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.641500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.641479    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqkw\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-kube-api-access-fcqkw\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.641725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.641705    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.734768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.734726    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.734948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.734792    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.734948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.734844    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.734953    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735042    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735084    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735112    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735139    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735154    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735171    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735290    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfls\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-kube-api-access-2hfls\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735470    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735500    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.735747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.735726    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.737387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.737366    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.737538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.737518    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.743376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.743350    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.743489 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.743473    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfls\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-kube-api-access-2hfls\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:38.818168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.818081    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:38.831984 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:38.831960    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:39.042809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.042777    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:39.122183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.122156    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"]
Nov 03 13:41:39.122728 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:41:39.122654    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902d255e_3bf6_42d2_a5ed_4a75f28d34b9.slice/crio-422b867e87990998a1522ffc573a55f1cf32a45826997b9c097d1af493312c8f WatchSource:0}: Error finding container 422b867e87990998a1522ffc573a55f1cf32a45826997b9c097d1af493312c8f: Status 404 returned error can't find the container with id 422b867e87990998a1522ffc573a55f1cf32a45826997b9c097d1af493312c8f
Nov 03 13:41:39.125246 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:41:39.125197    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ed9fb7_70f9_48f8_a648_3f73aa953dba.slice/crio-54ea33ce7f58b051d5d093cacf16636b76951b5fff27b0a12a53f8e42db3c6ee WatchSource:0}: Error finding container 54ea33ce7f58b051d5d093cacf16636b76951b5fff27b0a12a53f8e42db3c6ee: Status 404 returned error can't find the container with id 54ea33ce7f58b051d5d093cacf16636b76951b5fff27b0a12a53f8e42db3c6ee
Nov 03 13:41:39.125683 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.125650    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:41:39.125771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.125730    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:41:39.125836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.125773    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:41:39.127385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.127365    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"]
Nov 03 13:41:39.128001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.127970    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:41:39.128073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.128056    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:41:39.128133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.128113    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:41:39.803563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.801139    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:39.946487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.946458    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:40.009707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.992812    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" event={"ID":"22ed9fb7-70f9-48f8-a648-3f73aa953dba","Type":"ContainerStarted","Data":"a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408"}
Nov 03 13:41:40.009707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.992881    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" event={"ID":"22ed9fb7-70f9-48f8-a648-3f73aa953dba","Type":"ContainerStarted","Data":"54ea33ce7f58b051d5d093cacf16636b76951b5fff27b0a12a53f8e42db3c6ee"}
Nov 03 13:41:40.009707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.997170    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" event={"ID":"902d255e-3bf6-42d2-a5ed-4a75f28d34b9","Type":"ContainerStarted","Data":"814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f"}
Nov 03 13:41:40.009707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:39.997247    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" event={"ID":"902d255e-3bf6-42d2-a5ed-4a75f28d34b9","Type":"ContainerStarted","Data":"422b867e87990998a1522ffc573a55f1cf32a45826997b9c097d1af493312c8f"}
Nov 03 13:41:40.046230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:40.043556    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" podStartSLOduration=2.043532188 podStartE2EDuration="2.043532188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:40.031749742 +0000 UTC m=+1941.109514775" watchObservedRunningTime="2026-11-03 13:41:40.043532188 +0000 UTC m=+1941.121297217"
Nov 03 13:41:40.165852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:40.152493    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:40.818432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:40.818401    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:40.819759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:40.819734    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Nov 03 13:41:40.819915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:40.819778    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Nov 03 13:41:40.839590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:40.839560    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:40.854255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:40.850833    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 13:41:40.854255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:40.850899    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 13:41:40.873327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:40.870342    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:41.644152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:41.644061    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:41.819403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:41.819368    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Nov 03 13:41:41.819899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:41.819420    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Nov 03 13:41:41.832614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:41.832496    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 13:41:41.832614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:41.832555    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 13:41:42.479092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:42.479053    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:42.818603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:42.818526    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Nov 03 13:41:42.818603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:42.818583    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Nov 03 13:41:42.840234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:42.833249    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 13:41:42.840234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:42.833306    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 13:41:43.347237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:43.346705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:43.819103 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:43.819070    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Nov 03 13:41:43.819312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:43.819124    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Nov 03 13:41:43.847234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:43.846512    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 13:41:43.847234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:43.846571    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 13:41:44.179864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.179834    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:44.766235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.761497    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:41:44.809082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.808896    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" podStartSLOduration=6.80887562 podStartE2EDuration="6.80887562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:40.076877341 +0000 UTC m=+1941.154642436" watchObservedRunningTime="2026-11-03 13:41:44.80887562 +0000 UTC m=+1945.886640648"
Nov 03 13:41:44.819332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.818943    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Nov 03 13:41:44.819332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.818987    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Nov 03 13:41:44.883309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.882871    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 13:41:44.883309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.882923    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 13:41:44.893770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.890957    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:41:44.912075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.911984    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-plugins-config-volume\") pod \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\" (UID: \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\") "
Nov 03 13:41:44.912962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.912574    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-kube-api-access-htb7l\" (UniqueName: \"kubernetes.io/projected/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-kube-api-access-htb7l\") pod \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\" (UID: \"af2bd68c-986a-4fdc-aafd-d398b8b7d87e\") "
Nov 03 13:41:44.915605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.915562    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-plugins-config-volume" pod "af2bd68c-986a-4fdc-aafd-d398b8b7d87e" (UID: "af2bd68c-986a-4fdc-aafd-d398b8b7d87e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:44.916100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:44.916071    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-kube-api-access-htb7l" pod "af2bd68c-986a-4fdc-aafd-d398b8b7d87e" (UID: "af2bd68c-986a-4fdc-aafd-d398b8b7d87e"). InnerVolumeSpecName "kube-api-access-htb7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:45.015226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.015183    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-plugins-config-volume\") pod \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\" (UID: \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\") "
Nov 03 13:41:45.015396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.015370    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-kube-api-access-xwf4j\" (UniqueName: \"kubernetes.io/projected/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-kube-api-access-xwf4j\") pod \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\" (UID: \"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d\") "
Nov 03 13:41:45.016123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.016099    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:45.016175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.016129    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htb7l\" (UniqueName: \"kubernetes.io/projected/af2bd68c-986a-4fdc-aafd-d398b8b7d87e-kube-api-access-htb7l\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:45.016960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.016870    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-plugins-config-volume" pod "c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" (UID: "c22e671e-d5c5-4cc0-b357-74eddb4cbf7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:45.017785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.017762    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-kube-api-access-xwf4j" pod "c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" (UID: "c22e671e-d5c5-4cc0-b357-74eddb4cbf7d"). InnerVolumeSpecName "kube-api-access-xwf4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:45.092331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.092296    5420 generic.go:356] "Generic (PLEG): container finished" podID="af2bd68c-986a-4fdc-aafd-d398b8b7d87e" containerID="ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821" exitCode=0
Nov 03 13:41:45.092538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.092408    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"
Nov 03 13:41:45.092538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.092462    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs" event={"ID":"af2bd68c-986a-4fdc-aafd-d398b8b7d87e","Type":"ContainerDied","Data":"ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821"}
Nov 03 13:41:45.092538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.092508    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs" event={"ID":"af2bd68c-986a-4fdc-aafd-d398b8b7d87e","Type":"ContainerDied","Data":"a09d88cbf93ecc81a73bc5d1f3ba253d95bf37e3a2b5922f258871951960246a"}
Nov 03 13:41:45.092538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.092532    5420 scope.go:122] "RemoveContainer" containerID="ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821"
Nov 03 13:41:45.094505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.094480    5420 generic.go:356] "Generic (PLEG): container finished" podID="c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" containerID="41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee" exitCode=0
Nov 03 13:41:45.095264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.094544    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"
Nov 03 13:41:45.095369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.094638    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4" event={"ID":"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d","Type":"ContainerDied","Data":"41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee"}
Nov 03 13:41:45.095440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.095370    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4" event={"ID":"c22e671e-d5c5-4cc0-b357-74eddb4cbf7d","Type":"ContainerDied","Data":"d75e7997331cc884b48749057d3f617fa524bcf34d7d39109380554f0c818203"}
Nov 03 13:41:45.165436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.125049    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:45.165436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.125088    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwf4j\" (UniqueName: \"kubernetes.io/projected/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d-kube-api-access-xwf4j\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:45.165436 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.162051    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:45.208518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.208494    5420 scope.go:122] "RemoveContainer" containerID="ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821"
Nov 03 13:41:45.208865 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:45.208830    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821\": container with ID starting with ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821 not found: ID does not exist" containerID="ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821"
Nov 03 13:41:45.208975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.208896    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821"} err="failed to get container status \"ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821\": rpc error: code = NotFound desc = could not find container \"ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821\": container with ID starting with ad6d4258500ccf5f627152f6d8e17f2aaf4d4476f4a566ad7a435e37d3892821 not found: ID does not exist"
Nov 03 13:41:45.208975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.208927    5420 scope.go:122] "RemoveContainer" containerID="41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee"
Nov 03 13:41:45.210193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.210176    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"]
Nov 03 13:41:45.215010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.214977    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d3ca48-8f8dc6499-vkmf4"]
Nov 03 13:41:45.247778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.246423    5420 scope.go:122] "RemoveContainer" containerID="41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee"
Nov 03 13:41:45.250472 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:45.250433    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee\": container with ID starting with 41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee not found: ID does not exist" containerID="41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee"
Nov 03 13:41:45.250743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.250710    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee"} err="failed to get container status \"41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee\": rpc error: code = NotFound desc = could not find container \"41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee\": container with ID starting with 41137095b89517836b6f04cb430a9565e12b859f570d42b70205a304c9cf64ee not found: ID does not exist"
Nov 03 13:41:45.279286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.277245    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"]
Nov 03 13:41:45.282653 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.282622    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d3ca48-75754d84c9-hj5cs"]
Nov 03 13:41:45.820377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.820324    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Nov 03 13:41:45.820698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.820673    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Nov 03 13:41:45.843235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.840467    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 13:41:45.843235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:45.840524    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 13:41:46.654475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:46.654435    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:46.819067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:46.819021    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Nov 03 13:41:46.819279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:46.819095    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Nov 03 13:41:46.833200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:46.833155    5420 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 13:41:46.833419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:46.833253    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 13:41:47.085926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:47.085884    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2bd68c-986a-4fdc-aafd-d398b8b7d87e" path="/var/lib/kubelet/pods/af2bd68c-986a-4fdc-aafd-d398b8b7d87e/volumes"
Nov 03 13:41:47.086462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:47.086441    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" path="/var/lib/kubelet/pods/c22e671e-d5c5-4cc0-b357-74eddb4cbf7d/volumes"
Nov 03 13:41:47.530851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:47.530816    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:47.884341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:47.884242    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:47.884959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:47.884803    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:47.885638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:47.885611    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:47.895359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:47.895337    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:47.895725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:47.895695    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:47.896510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:47.896493    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:48.038042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.038005    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"]
Nov 03 13:41:48.050137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.050103    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3729b590-vgb7v"]
Nov 03 13:41:48.115522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.115486    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"]
Nov 03 13:41:48.116519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.116481    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" containerName="payload-processing"
Nov 03 13:41:48.116689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.116664    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" containerName="payload-processing"
Nov 03 13:41:48.116827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.116756    5420 cpu_manager.go:414] "containerMap: removing container" podUID="af2bd68c-986a-4fdc-aafd-d398b8b7d87e" containerName="payload-pre-processing"
Nov 03 13:41:48.116827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.116766    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2bd68c-986a-4fdc-aafd-d398b8b7d87e" containerName="payload-pre-processing"
Nov 03 13:41:48.116963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.116933    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2bd68c-986a-4fdc-aafd-d398b8b7d87e" containerName="payload-pre-processing"
Nov 03 13:41:48.116963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.116953    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22e671e-d5c5-4cc0-b357-74eddb4cbf7d" containerName="payload-processing"
Nov 03 13:41:48.136323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.125756    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.136323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.131840    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"e2e-conflict-a-9d13b16c-kserve-self-signed-certs\""
Nov 03 13:41:48.136323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.136022    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"]
Nov 03 13:41:48.199682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.198338    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-model-cache\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.199682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.198433    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-home\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.199682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.198509    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-tmp-dir\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.199682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.198563    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311540d8-1748-4474-8ec4-19904e354682-tls-certs\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.199682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.198618    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-dshm\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.199682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.198783    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q45fg\" (UniqueName: \"kubernetes.io/projected/311540d8-1748-4474-8ec4-19904e354682-kube-api-access-q45fg\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.214262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.214194    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:48.300114 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.300079    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-tmp-dir\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.300435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.300409    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311540d8-1748-4474-8ec4-19904e354682-tls-certs\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.300627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.300609    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-dshm\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.300881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.300860    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45fg\" (UniqueName: \"kubernetes.io/projected/311540d8-1748-4474-8ec4-19904e354682-kube-api-access-q45fg\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.301122 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.301104    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-model-cache\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.301316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.301297    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-home\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.301959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.301890    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-home\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.302399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.302376    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-tmp-dir\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.315246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.315202    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-model-cache\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.424308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.424096    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-dshm\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.430454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.424779    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q45fg\" (UniqueName: \"kubernetes.io/projected/311540d8-1748-4474-8ec4-19904e354682-kube-api-access-q45fg\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.431516 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.431398    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311540d8-1748-4474-8ec4-19904e354682-tls-certs\") pod \"e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.432440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.432368    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:48.462760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.462507    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:48.860257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:48.860228    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"]
Nov 03 13:41:48.860396 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:41:48.860318    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod311540d8_1748_4474_8ec4_19904e354682.slice/crio-e6980f08b001b15a53125cdaacb531909aaff729302a633fa5f06075a7e28294 WatchSource:0}: Error finding container e6980f08b001b15a53125cdaacb531909aaff729302a633fa5f06075a7e28294: Status 404 returned error can't find the container with id e6980f08b001b15a53125cdaacb531909aaff729302a633fa5f06075a7e28294
Nov 03 13:41:49.085203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:49.085158    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0b1f42-4664-4408-89c6-056f7458788e" path="/var/lib/kubelet/pods/7e0b1f42-4664-4408-89c6-056f7458788e/volumes"
Nov 03 13:41:49.390676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:49.390635    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz" event={"ID":"311540d8-1748-4474-8ec4-19904e354682","Type":"ContainerStarted","Data":"51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2"}
Nov 03 13:41:49.390855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:49.390685    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz" event={"ID":"311540d8-1748-4474-8ec4-19904e354682","Type":"ContainerStarted","Data":"e6980f08b001b15a53125cdaacb531909aaff729302a633fa5f06075a7e28294"}
Nov 03 13:41:49.429069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:49.428995    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz" podStartSLOduration=1.42897349 podStartE2EDuration="1.42897349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:49.421943 +0000 UTC m=+1950.499708210" watchObservedRunningTime="2026-11-03 13:41:49.42897349 +0000 UTC m=+1950.506738517"
Nov 03 13:41:49.445985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:49.445949    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:49.464026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:49.464000    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:50.302236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:50.292764    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:50.997188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:50.997155    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:51.164937 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:51.164909    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:51.688200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:51.688159    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:51.845647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:51.845610    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:52.491637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:52.491596    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:53.073104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.073055    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:53.472916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.472798    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:53.728667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.728573    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"]
Nov 03 13:41:53.729228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.729177    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy" containerID="cri-o://4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea" gracePeriod=30
Nov 03 13:41:53.743644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.743598    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"]
Nov 03 13:41:53.744141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.744091    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy" containerID="cri-o://d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe" gracePeriod=30
Nov 03 13:41:53.752113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.752083    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"]
Nov 03 13:41:53.752423 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.752371    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy" containerID="cri-o://814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f" gracePeriod=30
Nov 03 13:41:53.761701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.761666    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"]
Nov 03 13:41:53.761968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.761924    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy" containerID="cri-o://a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408" gracePeriod=30
Nov 03 13:41:53.834710 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.834668    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:53.992371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:53.983377    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:54.539333 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:54.539295    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:54.733238 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:54.733192    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:41:55.511129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:55.511095    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:56.224233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:56.224181    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:56.910738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:56.906750    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:57.697203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:57.697159    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:58.203435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:58.203400    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:58.368808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:58.368776    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:58.463658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:58.463580    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:58.479165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:58.479130    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:58.490456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:58.490431    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:41:58.515260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:58.515062    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:59.051854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.051827    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:41:59.099577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.098481    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:41:59.165424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.165259    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:59.179921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.179873    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:59.192452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.192428    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:59.195556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195531    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-envoy\") pod \"829b972d-478f-49e8-a7eb-678c3db13df8\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") "
Nov 03 13:41:59.195690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195564    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-data\") pod \"b513df0e-7f3b-4d73-85c8-e7a7830af016\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") "
Nov 03 13:41:59.195690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195587    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-envoy\") pod \"b513df0e-7f3b-4d73-85c8-e7a7830af016\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") "
Nov 03 13:41:59.195690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195636    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-token\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-token\") pod \"b513df0e-7f3b-4d73-85c8-e7a7830af016\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") "
Nov 03 13:41:59.195690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195654    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-socket\") pod \"829b972d-478f-49e8-a7eb-678c3db13df8\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") "
Nov 03 13:41:59.195690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195690    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-certs\") pod \"b513df0e-7f3b-4d73-85c8-e7a7830af016\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") "
Nov 03 13:41:59.195973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195708    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-credential-socket\") pod \"829b972d-478f-49e8-a7eb-678c3db13df8\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") "
Nov 03 13:41:59.195973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195750    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-istio-token\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-istio-token\") pod \"829b972d-478f-49e8-a7eb-678c3db13df8\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") "
Nov 03 13:41:59.195973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195773    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b513df0e-7f3b-4d73-85c8-e7a7830af016-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b513df0e-7f3b-4d73-85c8-e7a7830af016-istiod-ca-cert\") pod \"b513df0e-7f3b-4d73-85c8-e7a7830af016\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") "
Nov 03 13:41:59.195973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195837    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-certs\") pod \"829b972d-478f-49e8-a7eb-678c3db13df8\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") "
Nov 03 13:41:59.195973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195854    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-data" pod "b513df0e-7f3b-4d73-85c8-e7a7830af016" (UID: "b513df0e-7f3b-4d73-85c8-e7a7830af016"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.195973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195874    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-kube-api-access-wwjzq\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-kube-api-access-wwjzq\") pod \"b513df0e-7f3b-4d73-85c8-e7a7830af016\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") "
Nov 03 13:41:59.195973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195907    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-socket\") pod \"b513df0e-7f3b-4d73-85c8-e7a7830af016\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") "
Nov 03 13:41:59.195973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.195966    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-credential-socket\") pod \"b513df0e-7f3b-4d73-85c8-e7a7830af016\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") "
Nov 03 13:41:59.196519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.196063    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829b972d-478f-49e8-a7eb-678c3db13df8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829b972d-478f-49e8-a7eb-678c3db13df8-istiod-ca-cert\") pod \"829b972d-478f-49e8-a7eb-678c3db13df8\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") "
Nov 03 13:41:59.196519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.196187    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-credential-socket" pod "829b972d-478f-49e8-a7eb-678c3db13df8" (UID: "829b972d-478f-49e8-a7eb-678c3db13df8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.196519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.196272    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b513df0e-7f3b-4d73-85c8-e7a7830af016-istiod-ca-cert" pod "b513df0e-7f3b-4d73-85c8-e7a7830af016" (UID: "b513df0e-7f3b-4d73-85c8-e7a7830af016"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:59.196519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.196499    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-certs" pod "b513df0e-7f3b-4d73-85c8-e7a7830af016" (UID: "b513df0e-7f3b-4d73-85c8-e7a7830af016"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.196752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.196554    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-socket" pod "b513df0e-7f3b-4d73-85c8-e7a7830af016" (UID: "b513df0e-7f3b-4d73-85c8-e7a7830af016"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.196752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.196653    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-credential-socket" pod "b513df0e-7f3b-4d73-85c8-e7a7830af016" (UID: "b513df0e-7f3b-4d73-85c8-e7a7830af016"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.196867 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.196773    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-certs" pod "829b972d-478f-49e8-a7eb-678c3db13df8" (UID: "829b972d-478f-49e8-a7eb-678c3db13df8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.196950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.196925    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-socket" pod "829b972d-478f-49e8-a7eb-678c3db13df8" (UID: "829b972d-478f-49e8-a7eb-678c3db13df8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.197079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.196978    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-kube-api-access-8vn2x\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-kube-api-access-8vn2x\") pod \"829b972d-478f-49e8-a7eb-678c3db13df8\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") "
Nov 03 13:41:59.197079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197004    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829b972d-478f-49e8-a7eb-678c3db13df8-istiod-ca-cert" pod "829b972d-478f-49e8-a7eb-678c3db13df8" (UID: "829b972d-478f-49e8-a7eb-678c3db13df8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:59.197079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197032    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-data\") pod \"829b972d-478f-49e8-a7eb-678c3db13df8\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") "
Nov 03 13:41:59.197079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197072    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-podinfo\") pod \"b513df0e-7f3b-4d73-85c8-e7a7830af016\" (UID: \"b513df0e-7f3b-4d73-85c8-e7a7830af016\") "
Nov 03 13:41:59.197369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197142    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/829b972d-478f-49e8-a7eb-678c3db13df8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829b972d-478f-49e8-a7eb-678c3db13df8-istio-podinfo\") pod \"829b972d-478f-49e8-a7eb-678c3db13df8\" (UID: \"829b972d-478f-49e8-a7eb-678c3db13df8\") "
Nov 03 13:41:59.197369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197357    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-data" pod "829b972d-478f-49e8-a7eb-678c3db13df8" (UID: "829b972d-478f-49e8-a7eb-678c3db13df8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.197861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197838    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829b972d-478f-49e8-a7eb-678c3db13df8-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.197953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197869    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.197953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197886    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.197953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197903    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.197953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197917    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.197953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197931    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.197953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197945    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b513df0e-7f3b-4d73-85c8-e7a7830af016-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.198339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197959    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.198339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197974    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.198339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.197987    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.199639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.199582    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-istio-token" pod "829b972d-478f-49e8-a7eb-678c3db13df8" (UID: "829b972d-478f-49e8-a7eb-678c3db13df8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.199639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.199617    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-kube-api-access-8vn2x" pod "829b972d-478f-49e8-a7eb-678c3db13df8" (UID: "829b972d-478f-49e8-a7eb-678c3db13df8"). InnerVolumeSpecName "kube-api-access-8vn2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.199981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.199951    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-podinfo" pod "b513df0e-7f3b-4d73-85c8-e7a7830af016" (UID: "b513df0e-7f3b-4d73-85c8-e7a7830af016"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:41:59.200272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.200249    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-envoy" pod "829b972d-478f-49e8-a7eb-678c3db13df8" (UID: "829b972d-478f-49e8-a7eb-678c3db13df8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.200373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.200333    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-token" pod "b513df0e-7f3b-4d73-85c8-e7a7830af016" (UID: "b513df0e-7f3b-4d73-85c8-e7a7830af016"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.201158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.201134    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-kube-api-access-wwjzq" pod "b513df0e-7f3b-4d73-85c8-e7a7830af016" (UID: "b513df0e-7f3b-4d73-85c8-e7a7830af016"). InnerVolumeSpecName "kube-api-access-wwjzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.201639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.201614    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-envoy" pod "b513df0e-7f3b-4d73-85c8-e7a7830af016" (UID: "b513df0e-7f3b-4d73-85c8-e7a7830af016"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.201722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.201639    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/829b972d-478f-49e8-a7eb-678c3db13df8-istio-podinfo" pod "829b972d-478f-49e8-a7eb-678c3db13df8" (UID: "829b972d-478f-49e8-a7eb-678c3db13df8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:41:59.299017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.298922    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-podinfo\") pod \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") "
Nov 03 13:41:59.299017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.298985    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-data\") pod \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") "
Nov 03 13:41:59.299017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299015    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-certs\") pod \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") "
Nov 03 13:41:59.299351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299047    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-envoy\") pod \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") "
Nov 03 13:41:59.299351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299081    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-token\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-token\") pod \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") "
Nov 03 13:41:59.299351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299127    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-podinfo\") pod \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") "
Nov 03 13:41:59.299351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299159    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-credential-socket\") pod \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") "
Nov 03 13:41:59.299351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299188    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-certs\") pod \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") "
Nov 03 13:41:59.299351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299234    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-envoy\") pod \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") "
Nov 03 13:41:59.299351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299282    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-credential-socket\") pod \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") "
Nov 03 13:41:59.299351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299341    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-token\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-token\") pod \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") "
Nov 03 13:41:59.299812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299398    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-kube-api-access-fcqkw\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-kube-api-access-fcqkw\") pod \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") "
Nov 03 13:41:59.299812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299402    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-data" pod "902d255e-3bf6-42d2-a5ed-4a75f28d34b9" (UID: "902d255e-3bf6-42d2-a5ed-4a75f28d34b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.299812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299433    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-socket\") pod \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") "
Nov 03 13:41:59.299812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299544    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istiod-ca-cert\") pod \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") "
Nov 03 13:41:59.299812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299616    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istiod-ca-cert\") pod \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") "
Nov 03 13:41:59.299812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299662    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-socket\") pod \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\" (UID: \"902d255e-3bf6-42d2-a5ed-4a75f28d34b9\") "
Nov 03 13:41:59.299812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299707    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-kube-api-access-2hfls\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-kube-api-access-2hfls\") pod \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") "
Nov 03 13:41:59.299812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.299807    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-data\") pod \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\" (UID: \"22ed9fb7-70f9-48f8-a648-3f73aa953dba\") "
Nov 03 13:41:59.300316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300273    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-socket" pod "22ed9fb7-70f9-48f8-a648-3f73aa953dba" (UID: "22ed9fb7-70f9-48f8-a648-3f73aa953dba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300460    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-certs" pod "22ed9fb7-70f9-48f8-a648-3f73aa953dba" (UID: "22ed9fb7-70f9-48f8-a648-3f73aa953dba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300475    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-credential-socket" pod "22ed9fb7-70f9-48f8-a648-3f73aa953dba" (UID: "22ed9fb7-70f9-48f8-a648-3f73aa953dba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300609    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829b972d-478f-49e8-a7eb-678c3db13df8-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300631    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829b972d-478f-49e8-a7eb-678c3db13df8-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300681    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300695    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300710    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300709    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-credential-socket" pod "902d255e-3bf6-42d2-a5ed-4a75f28d34b9" (UID: "902d255e-3bf6-42d2-a5ed-4a75f28d34b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300724    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300739    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwjzq\" (UniqueName: \"kubernetes.io/projected/b513df0e-7f3b-4d73-85c8-e7a7830af016-kube-api-access-wwjzq\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300755    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300768    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300783    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vn2x\" (UniqueName: \"kubernetes.io/projected/829b972d-478f-49e8-a7eb-678c3db13df8-kube-api-access-8vn2x\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300799    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300798    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istiod-ca-cert" pod "902d255e-3bf6-42d2-a5ed-4a75f28d34b9" (UID: "902d255e-3bf6-42d2-a5ed-4a75f28d34b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:59.301128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.300814    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b513df0e-7f3b-4d73-85c8-e7a7830af016-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.302124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.301142    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istiod-ca-cert" pod "22ed9fb7-70f9-48f8-a648-3f73aa953dba" (UID: "22ed9fb7-70f9-48f8-a648-3f73aa953dba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:41:59.302124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.301407    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-socket" pod "902d255e-3bf6-42d2-a5ed-4a75f28d34b9" (UID: "902d255e-3bf6-42d2-a5ed-4a75f28d34b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.302124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.302112    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-certs" pod "902d255e-3bf6-42d2-a5ed-4a75f28d34b9" (UID: "902d255e-3bf6-42d2-a5ed-4a75f28d34b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.302413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.302366    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-envoy" pod "22ed9fb7-70f9-48f8-a648-3f73aa953dba" (UID: "22ed9fb7-70f9-48f8-a648-3f73aa953dba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.302413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.302375    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-data" pod "22ed9fb7-70f9-48f8-a648-3f73aa953dba" (UID: "22ed9fb7-70f9-48f8-a648-3f73aa953dba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.303891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.303843    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-kube-api-access-2hfls" pod "22ed9fb7-70f9-48f8-a648-3f73aa953dba" (UID: "22ed9fb7-70f9-48f8-a648-3f73aa953dba"). InnerVolumeSpecName "kube-api-access-2hfls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.304029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.304001    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-podinfo" pod "902d255e-3bf6-42d2-a5ed-4a75f28d34b9" (UID: "902d255e-3bf6-42d2-a5ed-4a75f28d34b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:41:59.304125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.304045    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-kube-api-access-fcqkw" pod "902d255e-3bf6-42d2-a5ed-4a75f28d34b9" (UID: "902d255e-3bf6-42d2-a5ed-4a75f28d34b9"). InnerVolumeSpecName "kube-api-access-fcqkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.304193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.304155    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-token" pod "902d255e-3bf6-42d2-a5ed-4a75f28d34b9" (UID: "902d255e-3bf6-42d2-a5ed-4a75f28d34b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.304312 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.304261    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-envoy" pod "902d255e-3bf6-42d2-a5ed-4a75f28d34b9" (UID: "902d255e-3bf6-42d2-a5ed-4a75f28d34b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:41:59.304685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.304653    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-podinfo" pod "22ed9fb7-70f9-48f8-a648-3f73aa953dba" (UID: "22ed9fb7-70f9-48f8-a648-3f73aa953dba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:41:59.305464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.305447    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-token" pod "22ed9fb7-70f9-48f8-a648-3f73aa953dba" (UID: "22ed9fb7-70f9-48f8-a648-3f73aa953dba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:41:59.402345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402311    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402343    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402353    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402362    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402370    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402379    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402387    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402396    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402404    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402412    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcqkw\" (UniqueName: \"kubernetes.io/projected/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-kube-api-access-fcqkw\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402421    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402430    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed9fb7-70f9-48f8-a648-3f73aa953dba-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402439    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902d255e-3bf6-42d2-a5ed-4a75f28d34b9-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.402606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.402448    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hfls\" (UniqueName: \"kubernetes.io/projected/22ed9fb7-70f9-48f8-a648-3f73aa953dba-kube-api-access-2hfls\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:41:59.486976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.486940    5420 generic.go:356] "Generic (PLEG): container finished" podID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerID="814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f" exitCode=0
Nov 03 13:41:59.487181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.486981    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" event={"ID":"902d255e-3bf6-42d2-a5ed-4a75f28d34b9","Type":"ContainerDied","Data":"814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f"}
Nov 03 13:41:59.487181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.487008    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"
Nov 03 13:41:59.487181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.487049    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk" event={"ID":"902d255e-3bf6-42d2-a5ed-4a75f28d34b9","Type":"ContainerDied","Data":"422b867e87990998a1522ffc573a55f1cf32a45826997b9c097d1af493312c8f"}
Nov 03 13:41:59.487181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.487079    5420 scope.go:122] "RemoveContainer" containerID="814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f"
Nov 03 13:41:59.488721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.488697    5420 generic.go:356] "Generic (PLEG): container finished" podID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerID="d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe" exitCode=0
Nov 03 13:41:59.488841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.488751    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"
Nov 03 13:41:59.488841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.488787    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" event={"ID":"b513df0e-7f3b-4d73-85c8-e7a7830af016","Type":"ContainerDied","Data":"d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe"}
Nov 03 13:41:59.488939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.488840    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk" event={"ID":"b513df0e-7f3b-4d73-85c8-e7a7830af016","Type":"ContainerDied","Data":"5f32c2babcc25aab7d6fb069bcabb4cbc3acbc171341d73f8ef447de980f5e8f"}
Nov 03 13:41:59.490439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.490414    5420 generic.go:356] "Generic (PLEG): container finished" podID="829b972d-478f-49e8-a7eb-678c3db13df8" containerID="4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea" exitCode=0
Nov 03 13:41:59.490552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.490462    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"
Nov 03 13:41:59.490635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.490617    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" event={"ID":"829b972d-478f-49e8-a7eb-678c3db13df8","Type":"ContainerDied","Data":"4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea"}
Nov 03 13:41:59.490694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.490654    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg" event={"ID":"829b972d-478f-49e8-a7eb-678c3db13df8","Type":"ContainerDied","Data":"c8a5af70a3c2984b3c81f3660fe36eda266650885005037d98e0911083e905b0"}
Nov 03 13:41:59.492412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.492386    5420 generic.go:356] "Generic (PLEG): container finished" podID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerID="a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408" exitCode=0
Nov 03 13:41:59.493077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.493028    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"
Nov 03 13:41:59.493186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.493111    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" event={"ID":"22ed9fb7-70f9-48f8-a648-3f73aa953dba","Type":"ContainerDied","Data":"a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408"}
Nov 03 13:41:59.493186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.493155    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc" event={"ID":"22ed9fb7-70f9-48f8-a648-3f73aa953dba","Type":"ContainerDied","Data":"54ea33ce7f58b051d5d093cacf16636b76951b5fff27b0a12a53f8e42db3c6ee"}
Nov 03 13:41:59.504836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.504815    5420 scope.go:122] "RemoveContainer" containerID="814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f"
Nov 03 13:41:59.505128 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:59.505097    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f\": container with ID starting with 814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f not found: ID does not exist" containerID="814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f"
Nov 03 13:41:59.505256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.505129    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f"} err="failed to get container status \"814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f\": rpc error: code = NotFound desc = could not find container \"814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f\": container with ID starting with 814b097d4a56081665ea96d5a0333870fa0af0f57262825cea3e0bc4f42eb45f not found: ID does not exist"
Nov 03 13:41:59.505256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.505152    5420 scope.go:122] "RemoveContainer" containerID="d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe"
Nov 03 13:41:59.514918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.514900    5420 scope.go:122] "RemoveContainer" containerID="d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe"
Nov 03 13:41:59.515158 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:59.515125    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe\": container with ID starting with d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe not found: ID does not exist" containerID="d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe"
Nov 03 13:41:59.515295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.515169    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe"} err="failed to get container status \"d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe\": rpc error: code = NotFound desc = could not find container \"d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe\": container with ID starting with d761ed4ee8fabca23b2421521a212cb13e523fd48ec772e060410838ac645abe not found: ID does not exist"
Nov 03 13:41:59.515295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.515190    5420 scope.go:122] "RemoveContainer" containerID="4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea"
Nov 03 13:41:59.525060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.524565    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"]
Nov 03 13:41:59.527672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.527652    5420 scope.go:122] "RemoveContainer" containerID="4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea"
Nov 03 13:41:59.527949 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:59.527933    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea\": container with ID starting with 4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea not found: ID does not exist" containerID="4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea"
Nov 03 13:41:59.528001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.527955    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea"} err="failed to get container status \"4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea\": rpc error: code = NotFound desc = could not find container \"4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea\": container with ID starting with 4a014938bf097b3abed56f1bf419796f9f7512f3fe6faf66035f3def82e531ea not found: ID does not exist"
Nov 03 13:41:59.528001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.527970    5420 scope.go:122] "RemoveContainer" containerID="a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408"
Nov 03 13:41:59.530281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.530201    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-72rxk"]
Nov 03 13:41:59.537220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.537194    5420 scope.go:122] "RemoveContainer" containerID="a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408"
Nov 03 13:41:59.537508 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:41:59.537491    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408\": container with ID starting with a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408 not found: ID does not exist" containerID="a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408"
Nov 03 13:41:59.537565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.537513    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408"} err="failed to get container status \"a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408\": rpc error: code = NotFound desc = could not find container \"a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408\": container with ID starting with a1e77913ff1b8449101b1d72aca851ab0f937f69c98cccf7785ef5426ddcf408 not found: ID does not exist"
Nov 03 13:41:59.548737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.548715    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"]
Nov 03 13:41:59.559405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.559348    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-bpzgk"]
Nov 03 13:41:59.585505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.585485    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"]
Nov 03 13:41:59.593667 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.593643    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-v7pkc"]
Nov 03 13:41:59.613770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.613749    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"]
Nov 03 13:41:59.621100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.621080    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-zcfgg"]
Nov 03 13:41:59.902295 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:41:59.899564    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:00.808730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:00.808691    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:01.036651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:01.036618    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"]
Nov 03 13:42:01.043739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:01.043710    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405a2ea2-xl97n"]
Nov 03 13:42:01.085400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:01.085337    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" path="/var/lib/kubelet/pods/22ed9fb7-70f9-48f8-a648-3f73aa953dba/volumes"
Nov 03 13:42:01.085781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:01.085767    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75736161-30da-4f0e-9e00-d5c951ea1161" path="/var/lib/kubelet/pods/75736161-30da-4f0e-9e00-d5c951ea1161/volumes"
Nov 03 13:42:01.086183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:01.086170    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" path="/var/lib/kubelet/pods/829b972d-478f-49e8-a7eb-678c3db13df8/volumes"
Nov 03 13:42:01.086921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:01.086907    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" path="/var/lib/kubelet/pods/902d255e-3bf6-42d2-a5ed-4a75f28d34b9/volumes"
Nov 03 13:42:01.087371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:01.087358    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" path="/var/lib/kubelet/pods/b513df0e-7f3b-4d73-85c8-e7a7830af016/volumes"
Nov 03 13:42:01.551350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:01.551311    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:02.384165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:02.384130    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:03.025559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:03.025517    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:03.651673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:03.651550    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:03.835645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:03.835602    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:04.237687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:04.237653    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"]
Nov 03 13:42:04.237884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:04.237864    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy" containerID="cri-o://4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497" gracePeriod=30
Nov 03 13:42:04.411919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:04.411886    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:05.043379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:05.036633    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:05.800183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:05.800136    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:05.987226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:05.987189    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:42:06.716655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:06.716618    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:07.634015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:07.633980    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:08.430237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:08.429382    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:09.128095 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.128056    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:09.498661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.498632    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:42:09.582510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.582471    5420 generic.go:356] "Generic (PLEG): container finished" podID="9a6ec136-db56-450a-81db-bdad382f9884" containerID="4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497" exitCode=0
Nov 03 13:42:09.582708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.582538    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"
Nov 03 13:42:09.582708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.582552    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" event={"ID":"9a6ec136-db56-450a-81db-bdad382f9884","Type":"ContainerDied","Data":"4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497"}
Nov 03 13:42:09.582708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.582610    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s" event={"ID":"9a6ec136-db56-450a-81db-bdad382f9884","Type":"ContainerDied","Data":"04cc8e525a313b5eba9ab8cbdee34fb2e90241b31d953e8b0b7f9f7183e6f2b8"}
Nov 03 13:42:09.582708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.582644    5420 scope.go:122] "RemoveContainer" containerID="4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497"
Nov 03 13:42:09.593944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.593925    5420 scope.go:122] "RemoveContainer" containerID="25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605"
Nov 03 13:42:09.599223 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599189    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a6ec136-db56-450a-81db-bdad382f9884-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6ec136-db56-450a-81db-bdad382f9884-istio-podinfo\") pod \"9a6ec136-db56-450a-81db-bdad382f9884\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") "
Nov 03 13:42:09.599337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599247    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-credential-socket\") pod \"9a6ec136-db56-450a-81db-bdad382f9884\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") "
Nov 03 13:42:09.599408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599363    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-certs\") pod \"9a6ec136-db56-450a-81db-bdad382f9884\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") "
Nov 03 13:42:09.599477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599442    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-kube-api-access-wqsvs\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-kube-api-access-wqsvs\") pod \"9a6ec136-db56-450a-81db-bdad382f9884\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") "
Nov 03 13:42:09.599540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599479    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6ec136-db56-450a-81db-bdad382f9884-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6ec136-db56-450a-81db-bdad382f9884-istiod-ca-cert\") pod \"9a6ec136-db56-450a-81db-bdad382f9884\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") "
Nov 03 13:42:09.599540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599514    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-credential-socket" pod "9a6ec136-db56-450a-81db-bdad382f9884" (UID: "9a6ec136-db56-450a-81db-bdad382f9884"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:09.599658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599610    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-envoy\") pod \"9a6ec136-db56-450a-81db-bdad382f9884\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") "
Nov 03 13:42:09.599658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599644    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-certs" pod "9a6ec136-db56-450a-81db-bdad382f9884" (UID: "9a6ec136-db56-450a-81db-bdad382f9884"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:09.599761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599683    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-socket\") pod \"9a6ec136-db56-450a-81db-bdad382f9884\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") "
Nov 03 13:42:09.599814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599774    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-istio-token\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-istio-token\") pod \"9a6ec136-db56-450a-81db-bdad382f9884\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") "
Nov 03 13:42:09.599883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599861    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-data\") pod \"9a6ec136-db56-450a-81db-bdad382f9884\" (UID: \"9a6ec136-db56-450a-81db-bdad382f9884\") "
Nov 03 13:42:09.599945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.599921    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6ec136-db56-450a-81db-bdad382f9884-istiod-ca-cert" pod "9a6ec136-db56-450a-81db-bdad382f9884" (UID: "9a6ec136-db56-450a-81db-bdad382f9884"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:09.600079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.600031    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-socket" pod "9a6ec136-db56-450a-81db-bdad382f9884" (UID: "9a6ec136-db56-450a-81db-bdad382f9884"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:09.600306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.600265    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-data" pod "9a6ec136-db56-450a-81db-bdad382f9884" (UID: "9a6ec136-db56-450a-81db-bdad382f9884"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:09.600528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.600508    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:09.600612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.600537    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:09.600612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.600553    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:09.600612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.600568    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:09.600612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.600583    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6ec136-db56-450a-81db-bdad382f9884-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:09.601840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.601818    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-envoy" pod "9a6ec136-db56-450a-81db-bdad382f9884" (UID: "9a6ec136-db56-450a-81db-bdad382f9884"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:09.601945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.601899    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a6ec136-db56-450a-81db-bdad382f9884-istio-podinfo" pod "9a6ec136-db56-450a-81db-bdad382f9884" (UID: "9a6ec136-db56-450a-81db-bdad382f9884"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:42:09.602017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.601992    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-kube-api-access-wqsvs" pod "9a6ec136-db56-450a-81db-bdad382f9884" (UID: "9a6ec136-db56-450a-81db-bdad382f9884"). InnerVolumeSpecName "kube-api-access-wqsvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:09.602161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.602145    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-istio-token" pod "9a6ec136-db56-450a-81db-bdad382f9884" (UID: "9a6ec136-db56-450a-81db-bdad382f9884"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:09.604086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.604068    5420 scope.go:122] "RemoveContainer" containerID="4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497"
Nov 03 13:42:09.604370 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:42:09.604350    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497\": container with ID starting with 4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497 not found: ID does not exist" containerID="4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497"
Nov 03 13:42:09.604450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.604378    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497"} err="failed to get container status \"4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497\": rpc error: code = NotFound desc = could not find container \"4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497\": container with ID starting with 4a2df9068b0a0e73bb4a5eaefb209f34154c8981dc84140f086c6ea4c55d3497 not found: ID does not exist"
Nov 03 13:42:09.604450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.604393    5420 scope.go:122] "RemoveContainer" containerID="25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605"
Nov 03 13:42:09.604648 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:42:09.604630    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605\": container with ID starting with 25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605 not found: ID does not exist" containerID="25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605"
Nov 03 13:42:09.604709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.604657    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605"} err="failed to get container status \"25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605\": rpc error: code = NotFound desc = could not find container \"25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605\": container with ID starting with 25528af6e720c70f7ae88ef1184640f00cf0df4b7c08bc0e19709ba540165605 not found: ID does not exist"
Nov 03 13:42:09.701675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.701603    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqsvs\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-kube-api-access-wqsvs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:09.701675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.701628    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6ec136-db56-450a-81db-bdad382f9884-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:09.701675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.701639    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6ec136-db56-450a-81db-bdad382f9884-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:09.701675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.701647    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6ec136-db56-450a-81db-bdad382f9884-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:09.808434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.805629    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:09.909057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.909022    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"]
Nov 03 13:42:09.913800 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:09.913769    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-rsk2s"]
Nov 03 13:42:10.419244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:10.418030    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:11.084811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:11.084783    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6ec136-db56-450a-81db-bdad382f9884" path="/var/lib/kubelet/pods/9a6ec136-db56-450a-81db-bdad382f9884/volumes"
Nov 03 13:42:11.172366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:11.172312    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:11.795248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:11.795192    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:12.391896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:12.391861    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:13.073767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:13.073733    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:13.729488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:13.727635    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:14.303469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:14.303434    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:15.005042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:15.004969    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:15.665588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:15.665562    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:16.274445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:16.274415    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:17.041040 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:17.040986    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:17.782907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:17.782870    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:18.339646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:18.339586    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:19.018105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:19.018059    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:19.554979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:19.554935    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:19.705400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:19.705352    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:20.401857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:20.401818    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:21.168490 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:21.168419    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:21.731852 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:21.731823    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"]
Nov 03 13:42:21.798627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:21.798597    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:22.068824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:22.068748    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:22.242091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:22.242058    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:22.496709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:22.496674    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:22.608705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:22.608678    5420 scope.go:122] "RemoveContainer" containerID="3b9507868d0b3669ceb5222ad6dc67dbb6270f37f9492ed0582658332cbce7d1"
Nov 03 13:42:22.618808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:22.618787    5420 scope.go:122] "RemoveContainer" containerID="c47e8b3a3d12a361b5e7c9dee13ca95b0ea62289b7b31cb0c11892ca3dcc6e36"
Nov 03 13:42:22.661784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:22.661757    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:22.806054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:22.805966    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:23.093622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.093550    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:23.268257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.266979    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:23.755650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.755615    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:23.914809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.914779    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"]
Nov 03 13:42:23.915413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915396    5420 cpu_manager.go:414] "containerMap: removing container" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy"
Nov 03 13:42:23.915491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915415    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy"
Nov 03 13:42:23.915491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915433    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy"
Nov 03 13:42:23.915491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915440    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy"
Nov 03 13:42:23.915491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915448    5420 cpu_manager.go:414] "containerMap: removing container" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy"
Nov 03 13:42:23.915491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915453    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy"
Nov 03 13:42:23.915491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915461    5420 cpu_manager.go:414] "containerMap: removing container" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy"
Nov 03 13:42:23.915491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915465    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy"
Nov 03 13:42:23.915491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915484    5420 cpu_manager.go:414] "containerMap: removing container" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy"
Nov 03 13:42:23.915491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915489    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy"
Nov 03 13:42:23.915787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915504    5420 cpu_manager.go:414] "containerMap: removing container" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy"
Nov 03 13:42:23.915787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915513    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy"
Nov 03 13:42:23.915787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915614    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ed9fb7-70f9-48f8-a648-3f73aa953dba" containerName="istio-proxy"
Nov 03 13:42:23.915787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915622    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b513df0e-7f3b-4d73-85c8-e7a7830af016" containerName="istio-proxy"
Nov 03 13:42:23.915787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915629    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="829b972d-478f-49e8-a7eb-678c3db13df8" containerName="istio-proxy"
Nov 03 13:42:23.915787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915635    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy"
Nov 03 13:42:23.915787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.915644    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="902d255e-3bf6-42d2-a5ed-4a75f28d34b9" containerName="istio-proxy"
Nov 03 13:42:23.921470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.921449    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:23.929571 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.929549    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ded926\""
Nov 03 13:42:23.929820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.929799    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"agc-cred-d534b8c80569\""
Nov 03 13:42:23.929820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.929799    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"openshift-service-ca.crt\""
Nov 03 13:42:23.932250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.930092    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"routing-overlay\""
Nov 03 13:42:23.932250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.930155    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"kube-root-ca.crt\""
Nov 03 13:42:23.932250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.931086    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"payload-processing-external-model-e2e--61c7371d-dockercfg-kphb2\""
Nov 03 13:42:23.932250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.931966    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:23.934878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.934856    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"]
Nov 03 13:42:24.001232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.991176    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.001232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.991404    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvwz\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-kube-api-access-brvwz\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.001232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.991477    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.001232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:23.991653    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.093390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.093289    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvwz\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-kube-api-access-brvwz\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.093390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.093384    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.093646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.093488    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.093712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.093653    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.094727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.094703    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.094857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.094810    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.097449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.097425    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.102985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.102962    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvwz\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-kube-api-access-brvwz\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-569cw928z\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.237173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.237141    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.536906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.536879    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"]
Nov 03 13:42:24.537057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.536951    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:24.537377 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:42:24.537350    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40070c5_a92a_4a71_96d9_12fdc971c2fd.slice/crio-8e94382b08ee5cc863d8792f81b27f13ca298a9b454862a8ec8b183a9fc6908d WatchSource:0}: Error finding container 8e94382b08ee5cc863d8792f81b27f13ca298a9b454862a8ec8b183a9fc6908d: Status 404 returned error can't find the container with id 8e94382b08ee5cc863d8792f81b27f13ca298a9b454862a8ec8b183a9fc6908d
Nov 03 13:42:24.731244 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:42:24.731173    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:42:24.752271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.752240    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z" event={"ID":"b40070c5-a92a-4a71-96d9-12fdc971c2fd","Type":"ContainerStarted","Data":"78ce78ad0c76d490165064a27ef5e1cd89d14174df53301c4d112c6a2fa0ff80"}
Nov 03 13:42:24.752426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.752293    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z" event={"ID":"b40070c5-a92a-4a71-96d9-12fdc971c2fd","Type":"ContainerStarted","Data":"8e94382b08ee5cc863d8792f81b27f13ca298a9b454862a8ec8b183a9fc6908d"}
Nov 03 13:42:24.752479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.752445    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:24.777443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:24.777401    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z" podStartSLOduration=1.777389533 podStartE2EDuration="1.777389533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:24.774907138 +0000 UTC m=+1985.852672175" watchObservedRunningTime="2026-11-03 13:42:24.777389533 +0000 UTC m=+1985.855154561"
Nov 03 13:42:25.308236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:25.304227    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:25.982974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:25.982940    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:26.861721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:26.861687    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:27.837021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:27.836988    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:28.160610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:28.160532    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:28.306765 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:28.306729    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:28.697787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:28.697750    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:29.296154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:29.296112    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:29.546813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:29.546744    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:30.028139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:30.028099    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:30.672347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:30.672315    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:31.313256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:31.313203    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:32.174523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:32.174493    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:32.680856 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:32.680817    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:33.102064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:33.102024    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:33.911917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:33.911869    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:34.377455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.377420    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"]
Nov 03 13:42:34.378148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.378132    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6ec136-db56-450a-81db-bdad382f9884" containerName="istio-proxy"
Nov 03 13:42:34.381885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.381861    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.393795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.393770    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"]
Nov 03 13:42:34.444881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.444846    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.445074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.444903    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.445074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.444987    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.445234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.445101    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.445234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.445130    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.445234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.445167    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.445427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.445328    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzgf8\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-kube-api-access-tzgf8\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.445492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.445466    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.445553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.445526    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd637f3-b50c-4f81-a105-c92294ad4e37-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.546956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.546926    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.546991    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzgf8\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-kube-api-access-tzgf8\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547115    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547170 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547156    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd637f3-b50c-4f81-a105-c92294ad4e37-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547226    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547287    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547317    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547407    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547444    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547447    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547538    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547780    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547845    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.547944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.547925    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd637f3-b50c-4f81-a105-c92294ad4e37-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.549787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.549757    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.549931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.549917    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.556302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.556281    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.556415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.556363    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:34.556662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.556644    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzgf8\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-kube-api-access-tzgf8\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.699692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.699607    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:34.988442 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.988417    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"]
Nov 03 13:42:34.989020 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:42:34.988999    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd637f3_b50c_4f81_a105_c92294ad4e37.slice/crio-7111c8042fa5d445eca1c611c4f2dd78bcd869436e348c7d5f7df4cd67c521ba WatchSource:0}: Error finding container 7111c8042fa5d445eca1c611c4f2dd78bcd869436e348c7d5f7df4cd67c521ba: Status 404 returned error can't find the container with id 7111c8042fa5d445eca1c611c4f2dd78bcd869436e348c7d5f7df4cd67c521ba
Nov 03 13:42:34.992027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.991989    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:42:34.992146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.992079    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:42:34.992146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:34.992127    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:42:35.119592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:35.116327    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:35.762747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:35.762719    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:42:35.797261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:35.797182    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:35.898257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:35.898202    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" event={"ID":"1dd637f3-b50c-4f81-a105-c92294ad4e37","Type":"ContainerStarted","Data":"73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e"}
Nov 03 13:42:35.898491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:35.898287    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" event={"ID":"1dd637f3-b50c-4f81-a105-c92294ad4e37","Type":"ContainerStarted","Data":"7111c8042fa5d445eca1c611c4f2dd78bcd869436e348c7d5f7df4cd67c521ba"}
Nov 03 13:42:35.947190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:35.946563    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" podStartSLOduration=1.946525982 podStartE2EDuration="1.946525982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:35.926768591 +0000 UTC m=+1997.004533662" watchObservedRunningTime="2026-11-03 13:42:35.946525982 +0000 UTC m=+1997.024291009"
Nov 03 13:42:35.991959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:35.991920    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:36.307740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.306982    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:36.592781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.592223    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"]
Nov 03 13:42:36.592781 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.592709    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy" containerID="cri-o://3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88" gracePeriod=30
Nov 03 13:42:36.598283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.597549    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"]
Nov 03 13:42:36.598283 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.597742    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy" containerID="cri-o://be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862" gracePeriod=30
Nov 03 13:42:36.699324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.699284    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:36.700311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.700287    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:36.701769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.701453    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 13:42:36.701769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.701519    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 13:42:36.763262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.759466    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz" podUID="311540d8-1748-4474-8ec4-19904e354682" containerName="main" containerID="cri-o://51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2" gracePeriod=45
Nov 03 13:42:36.863943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:36.863742    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:37.158102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.158076    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:42:37.191469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.191440    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-dshm\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-dshm\") pod \"311540d8-1748-4474-8ec4-19904e354682\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") "
Nov 03 13:42:37.191666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.191493    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311540d8-1748-4474-8ec4-19904e354682-kube-api-access-q45fg\" (UniqueName: \"kubernetes.io/projected/311540d8-1748-4474-8ec4-19904e354682-kube-api-access-q45fg\") pod \"311540d8-1748-4474-8ec4-19904e354682\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") "
Nov 03 13:42:37.191666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.191616    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-model-cache\") pod \"311540d8-1748-4474-8ec4-19904e354682\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") "
Nov 03 13:42:37.191666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.191651    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311540d8-1748-4474-8ec4-19904e354682-tls-certs\" (UniqueName: \"kubernetes.io/secret/311540d8-1748-4474-8ec4-19904e354682-tls-certs\") pod \"311540d8-1748-4474-8ec4-19904e354682\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") "
Nov 03 13:42:37.191833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.191739    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-home\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-home\") pod \"311540d8-1748-4474-8ec4-19904e354682\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") "
Nov 03 13:42:37.191833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.191815    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-tmp-dir\") pod \"311540d8-1748-4474-8ec4-19904e354682\" (UID: \"311540d8-1748-4474-8ec4-19904e354682\") "
Nov 03 13:42:37.192150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.192008    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-model-cache" pod "311540d8-1748-4474-8ec4-19904e354682" (UID: "311540d8-1748-4474-8ec4-19904e354682"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:37.192308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.192230    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-tmp-dir" pod "311540d8-1748-4474-8ec4-19904e354682" (UID: "311540d8-1748-4474-8ec4-19904e354682"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:37.192982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.192458    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-home" pod "311540d8-1748-4474-8ec4-19904e354682" (UID: "311540d8-1748-4474-8ec4-19904e354682"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:37.192982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.192666    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:37.192982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.192688    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:37.192982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.192704    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:37.194390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.194354    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-dshm" pod "311540d8-1748-4474-8ec4-19904e354682" (UID: "311540d8-1748-4474-8ec4-19904e354682"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:37.194626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.194596    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311540d8-1748-4474-8ec4-19904e354682-kube-api-access-q45fg" pod "311540d8-1748-4474-8ec4-19904e354682" (UID: "311540d8-1748-4474-8ec4-19904e354682"). InnerVolumeSpecName "kube-api-access-q45fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:37.194985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.194961    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311540d8-1748-4474-8ec4-19904e354682-tls-certs" pod "311540d8-1748-4474-8ec4-19904e354682" (UID: "311540d8-1748-4474-8ec4-19904e354682"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:42:37.293995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.293959    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311540d8-1748-4474-8ec4-19904e354682-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:37.293995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.293990    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311540d8-1748-4474-8ec4-19904e354682-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:37.293995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.294000    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q45fg\" (UniqueName: \"kubernetes.io/projected/311540d8-1748-4474-8ec4-19904e354682-kube-api-access-q45fg\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:37.459583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.459550    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:37.601110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.601080    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:37.700358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.700322    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 13:42:37.700612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.700380    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 13:42:37.945869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.945772    5420 generic.go:356] "Generic (PLEG): container finished" podID="311540d8-1748-4474-8ec4-19904e354682" containerID="51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2" exitCode=0
Nov 03 13:42:37.945869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.945836    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"
Nov 03 13:42:37.946134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.945858    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz" event={"ID":"311540d8-1748-4474-8ec4-19904e354682","Type":"ContainerDied","Data":"51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2"}
Nov 03 13:42:37.946134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.945915    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz" event={"ID":"311540d8-1748-4474-8ec4-19904e354682","Type":"ContainerDied","Data":"e6980f08b001b15a53125cdaacb531909aaff729302a633fa5f06075a7e28294"}
Nov 03 13:42:37.946134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.945938    5420 scope.go:122] "RemoveContainer" containerID="51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2"
Nov 03 13:42:37.970008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.969965    5420 scope.go:122] "RemoveContainer" containerID="51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2"
Nov 03 13:42:37.970874 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:42:37.970846    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2\": container with ID starting with 51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2 not found: ID does not exist" containerID="51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2"
Nov 03 13:42:37.971028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.971004    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2"} err="failed to get container status \"51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2\": rpc error: code = NotFound desc = could not find container \"51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2\": container with ID starting with 51484809d91b57e60221e9b06783812e15351f338a9da6597201e22ccab513b2 not found: ID does not exist"
Nov 03 13:42:37.982783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.982758    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"]
Nov 03 13:42:37.988118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:37.988091    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-a-9d13b16c-kserve-56fd6859bb-8kglz"]
Nov 03 13:42:38.289238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:38.281159    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:38.700113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:38.700033    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 13:42:38.700113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:38.700090    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 13:42:39.063419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:39.063382    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:39.086033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:39.086004    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311540d8-1748-4474-8ec4-19904e354682" path="/var/lib/kubelet/pods/311540d8-1748-4474-8ec4-19904e354682/volumes"
Nov 03 13:42:39.700778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:39.700744    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 13:42:39.701268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:39.700801    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 13:42:39.818874 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:39.818839    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:39.995198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:39.995163    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:42:40.495939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:40.495909    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:40.702238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:40.701183    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 13:42:40.702238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:40.701259    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 13:42:40.806109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:40.806024    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:42:40.806416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:40.806380    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:42:41.299748 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.299704    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:41.600358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.600267    5420 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:42:41.600358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.600341    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:42:41.700346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.700312    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 13:42:41.700525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.700367    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 13:42:41.922124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.922094    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:42:41.987978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.987945    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:42:41.995646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.995611    5420 generic.go:356] "Generic (PLEG): container finished" podID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerID="3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88" exitCode=0
Nov 03 13:42:41.995776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.995710    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"
Nov 03 13:42:41.995776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.995744    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" event={"ID":"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab","Type":"ContainerDied","Data":"3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88"}
Nov 03 13:42:41.995884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.995821    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h" event={"ID":"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab","Type":"ContainerDied","Data":"a734950490c13e17a11d76559a53c9517645aee4143bd888ce6ee20763d5a024"}
Nov 03 13:42:41.995884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.995845    5420 scope.go:122] "RemoveContainer" containerID="3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88"
Nov 03 13:42:41.997492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.997464    5420 generic.go:356] "Generic (PLEG): container finished" podID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerID="be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862" exitCode=0
Nov 03 13:42:41.997595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.997497    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"
Nov 03 13:42:41.997595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.997541    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" event={"ID":"cc4734f2-cf4e-41d1-a84f-8a606818d391","Type":"ContainerDied","Data":"be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862"}
Nov 03 13:42:41.997595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:41.997584    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw" event={"ID":"cc4734f2-cf4e-41d1-a84f-8a606818d391","Type":"ContainerDied","Data":"0f5e56cbc33a19b4e46cb36006aeb4348a54bbdec0f9bfb5b4c623cbfd09d969"}
Nov 03 13:42:42.009322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.009265    5420 scope.go:122] "RemoveContainer" containerID="3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88"
Nov 03 13:42:42.009585 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:42:42.009563    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88\": container with ID starting with 3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88 not found: ID does not exist" containerID="3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88"
Nov 03 13:42:42.009705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.009597    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88"} err="failed to get container status \"3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88\": rpc error: code = NotFound desc = could not find container \"3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88\": container with ID starting with 3848519a8c335ff833a960e9326251fc35b5f49bfbb0f5892bcca02d81956e88 not found: ID does not exist"
Nov 03 13:42:42.009705 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.009619    5420 scope.go:122] "RemoveContainer" containerID="be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862"
Nov 03 13:42:42.027651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.027629    5420 scope.go:122] "RemoveContainer" containerID="be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862"
Nov 03 13:42:42.027989 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:42:42.027969    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862\": container with ID starting with be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862 not found: ID does not exist" containerID="be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862"
Nov 03 13:42:42.028076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.028000    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862"} err="failed to get container status \"be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862\": rpc error: code = NotFound desc = could not find container \"be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862\": container with ID starting with be9d1553b30bcd43511101fcca2410a3b88a3dba33c66fecb0897d7395e78862 not found: ID does not exist"
Nov 03 13:42:42.094688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.094640    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-credential-socket\") pod \"cc4734f2-cf4e-41d1-a84f-8a606818d391\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") "
Nov 03 13:42:42.095027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095002    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-kube-api-access-6bwv6\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-kube-api-access-6bwv6\") pod \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") "
Nov 03 13:42:42.095183 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095165    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-podinfo\") pod \"cc4734f2-cf4e-41d1-a84f-8a606818d391\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") "
Nov 03 13:42:42.095417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095298    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-credential-socket" pod "cc4734f2-cf4e-41d1-a84f-8a606818d391" (UID: "cc4734f2-cf4e-41d1-a84f-8a606818d391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.095417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095387    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-socket\") pod \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") "
Nov 03 13:42:42.095617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095441    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-socket\") pod \"cc4734f2-cf4e-41d1-a84f-8a606818d391\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") "
Nov 03 13:42:42.095617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095506    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istiod-ca-cert\") pod \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") "
Nov 03 13:42:42.095617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095548    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-certs\") pod \"cc4734f2-cf4e-41d1-a84f-8a606818d391\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") "
Nov 03 13:42:42.095617 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095582    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-certs\") pod \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") "
Nov 03 13:42:42.095897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095635    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4734f2-cf4e-41d1-a84f-8a606818d391-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4734f2-cf4e-41d1-a84f-8a606818d391-istiod-ca-cert\") pod \"cc4734f2-cf4e-41d1-a84f-8a606818d391\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") "
Nov 03 13:42:42.095897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095719    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-socket" pod "cc4734f2-cf4e-41d1-a84f-8a606818d391" (UID: "cc4734f2-cf4e-41d1-a84f-8a606818d391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.095897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095708    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-socket" pod "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" (UID: "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.095897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095822    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-data\") pod \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") "
Nov 03 13:42:42.095897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095863    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-envoy\") pod \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") "
Nov 03 13:42:42.096200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095929    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-certs" pod "cc4734f2-cf4e-41d1-a84f-8a606818d391" (UID: "cc4734f2-cf4e-41d1-a84f-8a606818d391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.096200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095932    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-data\") pod \"cc4734f2-cf4e-41d1-a84f-8a606818d391\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") "
Nov 03 13:42:42.096200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096001    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-token\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-token\") pod \"cc4734f2-cf4e-41d1-a84f-8a606818d391\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") "
Nov 03 13:42:42.096414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096203    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-credential-socket\") pod \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") "
Nov 03 13:42:42.096414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.095942    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-certs" pod "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" (UID: "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.096414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096184    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-data" pod "cc4734f2-cf4e-41d1-a84f-8a606818d391" (UID: "cc4734f2-cf4e-41d1-a84f-8a606818d391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.096414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096221    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-data" pod "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" (UID: "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.096414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096240    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istiod-ca-cert" pod "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" (UID: "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:42.096716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096504    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-credential-socket" pod "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" (UID: "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.096716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096532    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4734f2-cf4e-41d1-a84f-8a606818d391-istiod-ca-cert" pod "cc4734f2-cf4e-41d1-a84f-8a606818d391" (UID: "cc4734f2-cf4e-41d1-a84f-8a606818d391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:42.096716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096637    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-envoy\") pod \"cc4734f2-cf4e-41d1-a84f-8a606818d391\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") "
Nov 03 13:42:42.096716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096683    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-podinfo\") pod \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") "
Nov 03 13:42:42.096716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096711    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-token\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-token\") pod \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\" (UID: \"4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab\") "
Nov 03 13:42:42.097008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.096753    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-kube-api-access-2qdlg\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-kube-api-access-2qdlg\") pod \"cc4734f2-cf4e-41d1-a84f-8a606818d391\" (UID: \"cc4734f2-cf4e-41d1-a84f-8a606818d391\") "
Nov 03 13:42:42.097472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097446    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.097578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097478    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.097578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097496    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4734f2-cf4e-41d1-a84f-8a606818d391-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.097578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097511    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.097578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097529    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.097578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097543    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.097578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097559    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.097578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097574    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.097934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097588    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.097934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097603    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.098057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.097934    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-kube-api-access-6bwv6" pod "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" (UID: "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab"). InnerVolumeSpecName "kube-api-access-6bwv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:42.098827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.098270    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-podinfo" pod "cc4734f2-cf4e-41d1-a84f-8a606818d391" (UID: "cc4734f2-cf4e-41d1-a84f-8a606818d391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:42:42.100498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.100353    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-envoy" pod "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" (UID: "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.100498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.100393    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-token" pod "cc4734f2-cf4e-41d1-a84f-8a606818d391" (UID: "cc4734f2-cf4e-41d1-a84f-8a606818d391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:42.101703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.100784    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-kube-api-access-2qdlg" pod "cc4734f2-cf4e-41d1-a84f-8a606818d391" (UID: "cc4734f2-cf4e-41d1-a84f-8a606818d391"). InnerVolumeSpecName "kube-api-access-2qdlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:42.101703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.101029    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-token" pod "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" (UID: "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:42.101703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.101541    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-podinfo" pod "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" (UID: "4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:42:42.103858 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.103577    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-envoy" pod "cc4734f2-cf4e-41d1-a84f-8a606818d391" (UID: "cc4734f2-cf4e-41d1-a84f-8a606818d391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:42.218385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.199470    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.218385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.199514    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.218385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.199532    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.218385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.199549    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.218385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.199562    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.218385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.199578    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qdlg\" (UniqueName: \"kubernetes.io/projected/cc4734f2-cf4e-41d1-a84f-8a606818d391-kube-api-access-2qdlg\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.218385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.199594    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bwv6\" (UniqueName: \"kubernetes.io/projected/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab-kube-api-access-6bwv6\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.218385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.199609    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4734f2-cf4e-41d1-a84f-8a606818d391-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:42.224917 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.224872    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:42.334823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.334783    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"]
Nov 03 13:42:42.338932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.338903    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-qft5h"]
Nov 03 13:42:42.351784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.351751    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"]
Nov 03 13:42:42.359075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.359043    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-q49zw"]
Nov 03 13:42:42.742090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.742062    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:42.742395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.742377    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:42.743275 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:42.743252    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:42:43.085721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:43.085691    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" path="/var/lib/kubelet/pods/4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab/volumes"
Nov 03 13:42:43.086251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:43.086203    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" path="/var/lib/kubelet/pods/cc4734f2-cf4e-41d1-a84f-8a606818d391/volumes"
Nov 03 13:42:43.163163 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:43.163130    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:44.020136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:44.020103    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:44.870458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:44.870423    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:45.122202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:45.122126    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:45.276859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:45.276824    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:45.773920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:45.773889    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:45.926651 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:45.926617    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:46.386092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:46.386062    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:46.618243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:46.618020    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:47.324061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:47.324022    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:47.894056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:47.894021    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:48.097469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:48.097439    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:48.707988 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:48.707947    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:49.312200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.312125    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:49.333984 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.333953    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"]
Nov 03 13:42:49.334899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.334868    5420 cpu_manager.go:414] "containerMap: removing container" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy"
Nov 03 13:42:49.334899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.334893    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy"
Nov 03 13:42:49.335080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.334916    5420 cpu_manager.go:414] "containerMap: removing container" podUID="311540d8-1748-4474-8ec4-19904e354682" containerName="main"
Nov 03 13:42:49.335080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.334922    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311540d8-1748-4474-8ec4-19904e354682" containerName="main"
Nov 03 13:42:49.335080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.334942    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy"
Nov 03 13:42:49.335080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.334949    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy"
Nov 03 13:42:49.335288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.335227    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="311540d8-1748-4474-8ec4-19904e354682" containerName="main"
Nov 03 13:42:49.335288 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.335255    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b831a0b-a2a2-4fcc-8cd6-74d45b48dfab" containerName="istio-proxy"
Nov 03 13:42:49.335363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.335291    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4734f2-cf4e-41d1-a84f-8a606818d391" containerName="istio-proxy"
Nov 03 13:42:49.347256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.347191    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.356650 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.356627    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"]
Nov 03 13:42:49.381905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.381847    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"]
Nov 03 13:42:49.389241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.387454    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.393967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.393937    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"]
Nov 03 13:42:49.407601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.407572    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqhgd\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-kube-api-access-pqhgd\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.407731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.407620    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.407813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.407762    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.408227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.408188    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.408399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.408362    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.408569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.408550    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.408719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.408702    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.408860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.408839    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.408986 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.408970    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.409100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.409086    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.409278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.409259    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.409405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.409385    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqft\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-kube-api-access-shqft\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.409504 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.409434    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.409564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.409536    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.409619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.409564    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.409619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.409610    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.409729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.409640    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.409729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.409681    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.510793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.510764    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.510793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.510797    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.511035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.510828    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.510947    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.511035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511005    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511054    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqhgd\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-kube-api-access-pqhgd\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511096    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511129    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511260    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.511498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511270    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511317    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.511498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511346    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.511498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511378    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511417    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511498 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511495    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511549    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.511875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511587    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511624    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.511875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511677    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511728    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.511875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511775    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.511875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511799    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.511875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511830    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqft\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-kube-api-access-shqft\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.512455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511880    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.512455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.511880    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.512455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.512035    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.512455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.512362    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.512685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.512658    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.513730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.513695    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.514175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.514149    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.514321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.514301    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.514383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.514367    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.522669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.522638    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.522830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.522812    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.523024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.523001    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqhgd\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-kube-api-access-pqhgd\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:49.523087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.523068    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqft\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-kube-api-access-shqft\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.669900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.669825    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:49.704349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:49.704322    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:50.011685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.011655    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:50.068227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.068188    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"]
Nov 03 13:42:50.069698 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:42:50.069671    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bab1e71_50d2_48a2_bdf7_c5f7a5bee42f.slice/crio-38c691e7476fb637649e97db52146b53f4d705819cb9bd30bb70eeebd3e4424c WatchSource:0}: Error finding container 38c691e7476fb637649e97db52146b53f4d705819cb9bd30bb70eeebd3e4424c: Status 404 returned error can't find the container with id 38c691e7476fb637649e97db52146b53f4d705819cb9bd30bb70eeebd3e4424c
Nov 03 13:42:50.072178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.072148    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:42:50.072251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.072227    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:42:50.072299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.072262    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:42:50.110015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.109988    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"]
Nov 03 13:42:50.110884 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:42:50.110863    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67dc242_8956_4153_9d6e_7cedaf3f5b73.slice/crio-6bc9fdcfcaea0b866481f411c94c7d3f8814664f8236e1c30d3cd30d49c018e3 WatchSource:0}: Error finding container 6bc9fdcfcaea0b866481f411c94c7d3f8814664f8236e1c30d3cd30d49c018e3: Status 404 returned error can't find the container with id 6bc9fdcfcaea0b866481f411c94c7d3f8814664f8236e1c30d3cd30d49c018e3
Nov 03 13:42:50.113487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.113457    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:42:50.113576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.113522    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:42:50.113576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.113551    5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:42:50.874238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:50.871872    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:51.084397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.084367    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" event={"ID":"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f","Type":"ContainerStarted","Data":"1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec"}
Nov 03 13:42:51.084770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.084411    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" event={"ID":"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f","Type":"ContainerStarted","Data":"38c691e7476fb637649e97db52146b53f4d705819cb9bd30bb70eeebd3e4424c"}
Nov 03 13:42:51.084770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.084427    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" event={"ID":"d67dc242-8956-4153-9d6e-7cedaf3f5b73","Type":"ContainerStarted","Data":"56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0"}
Nov 03 13:42:51.084770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.084442    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" event={"ID":"d67dc242-8956-4153-9d6e-7cedaf3f5b73","Type":"ContainerStarted","Data":"6bc9fdcfcaea0b866481f411c94c7d3f8814664f8236e1c30d3cd30d49c018e3"}
Nov 03 13:42:51.101001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.100959    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" podStartSLOduration=2.10094537 podStartE2EDuration="2.10094537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:51.100097893 +0000 UTC m=+2012.177862921" watchObservedRunningTime="2026-11-03 13:42:51.10094537 +0000 UTC m=+2012.178710396"
Nov 03 13:42:51.164294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.163286    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" podStartSLOduration=2.163265186 podStartE2EDuration="2.163265186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:51.140828813 +0000 UTC m=+2012.218593840" watchObservedRunningTime="2026-11-03 13:42:51.163265186 +0000 UTC m=+2012.241030214"
Nov 03 13:42:51.572241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.570821    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:51.671191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.671042    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:51.672979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.672952    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 13:42:51.673140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.673015    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 13:42:51.704784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.704757    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:51.706102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.706074    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 03 13:42:51.706248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.706124    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 03 13:42:51.903577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:51.903431    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:52.062231 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.061609    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:52.252237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.242126    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:52.442235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.434329    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:52.673381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.670634    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 13:42:52.673381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.670699    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 13:42:52.705251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.704864    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 03 13:42:52.705251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.704909    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 03 13:42:52.761747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.761712    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"]
Nov 03 13:42:52.762199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.762166    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy" containerID="cri-o://3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1" gracePeriod=30
Nov 03 13:42:52.805353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:52.805325    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:53.231238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:53.225671    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:53.619514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:53.619428    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:53.672237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:53.671678    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 13:42:53.672237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:53.671740    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 13:42:53.705310 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:53.705278    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 03 13:42:53.705484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:53.705329    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 03 13:42:53.960292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:53.946963    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:54.619377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:54.619344    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:54.677238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:54.671612    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 13:42:54.677238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:54.671672    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 13:42:54.705027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:54.704984    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 03 13:42:54.705252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:54.705030    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 03 13:42:54.778272 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:42:54.778237    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:42:55.412881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:55.412845    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:55.670733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:55.670643    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 13:42:55.670944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:55.670916    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 13:42:55.705185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:55.705152    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 03 13:42:55.705619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:55.705238    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 03 13:42:56.119202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:56.119169    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:56.207006 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:56.206971    5420 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:42:56.207175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:56.207032    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:42:56.671263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:56.671197    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 13:42:56.671445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:56.671306    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 13:42:56.704761 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:56.704727    5420 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 03 13:42:56.704935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:56.704787    5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 03 13:42:56.804886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:56.804851    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:57.408521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:57.408448    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:57.675364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:57.675289    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:57.675768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:57.675750    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:57.676458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:57.676439    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:42:57.708637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:57.708612    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:57.708903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:57.708886    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:57.709649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:57.709631    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:42:58.077106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.077085    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:42:58.115346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.115292    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:58.144947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.144915    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-credential-socket\") pod \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") "
Nov 03 13:42:58.145148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.144983    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-certs\") pod \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") "
Nov 03 13:42:58.145148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145025    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istiod-ca-cert\") pod \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") "
Nov 03 13:42:58.145148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145119    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-kube-api-access-djk7r\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-kube-api-access-djk7r\") pod \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") "
Nov 03 13:42:58.145371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145235    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-envoy\") pod \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") "
Nov 03 13:42:58.145371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145269    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-data\") pod \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") "
Nov 03 13:42:58.145371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145270    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-credential-socket" pod "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" (UID: "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.145371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145312    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-token\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-token\") pod \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") "
Nov 03 13:42:58.145611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145410    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-certs" pod "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" (UID: "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.145611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145421    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-podinfo\") pod \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") "
Nov 03 13:42:58.145611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145487    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-socket\") pod \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\" (UID: \"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f\") "
Nov 03 13:42:58.145611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145532    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istiod-ca-cert" pod "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" (UID: "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:58.145846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.145627    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-data" pod "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" (UID: "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.146085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.146062    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.146178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.146089    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.146178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.146104    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.146178 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.146118    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.146393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.146273    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-socket" pod "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" (UID: "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.148096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.148058    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-token" pod "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" (UID: "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:58.148597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.148567    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-podinfo" pod "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" (UID: "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:42:58.148693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.148653    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-kube-api-access-djk7r" pod "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" (UID: "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f"). InnerVolumeSpecName "kube-api-access-djk7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:58.149016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.148990    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-envoy" pod "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" (UID: "f02f66f5-b9f5-4cf4-bb36-dbf981efb16f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:42:58.161634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.161608    5420 generic.go:356] "Generic (PLEG): container finished" podID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerID="3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1" exitCode=0
Nov 03 13:42:58.161766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.161664    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"
Nov 03 13:42:58.161766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.161696    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" event={"ID":"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f","Type":"ContainerDied","Data":"3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1"}
Nov 03 13:42:58.161766 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.161757    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h" event={"ID":"f02f66f5-b9f5-4cf4-bb36-dbf981efb16f","Type":"ContainerDied","Data":"15375beb2f4cd19da26990a0c34bf2e35f6ba1862bc1f8ed2cb1323fd26bfd36"}
Nov 03 13:42:58.161946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.161787    5420 scope.go:122] "RemoveContainer" containerID="3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1"
Nov 03 13:42:58.185240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.185188    5420 scope.go:122] "RemoveContainer" containerID="3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1"
Nov 03 13:42:58.185543 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:42:58.185523    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1\": container with ID starting with 3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1 not found: ID does not exist" containerID="3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1"
Nov 03 13:42:58.185614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.185550    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1"} err="failed to get container status \"3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1\": rpc error: code = NotFound desc = could not find container \"3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1\": container with ID starting with 3419ba9eeeae8fd75b3f41a9410ec3a02f09ee1380e3ee876774e1d67c666ae1 not found: ID does not exist"
Nov 03 13:42:58.194540 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.194517    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"]
Nov 03 13:42:58.200974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.200950    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-jhb8h"]
Nov 03 13:42:58.248060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.247995    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.248060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.248022    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.248060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.248033    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.248060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.248042    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djk7r\" (UniqueName: \"kubernetes.io/projected/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-kube-api-access-djk7r\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.248060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.248052    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:58.727637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.727591    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:58.896927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.896896    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"]
Nov 03 13:42:58.906773 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:58.906750    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"]
Nov 03 13:42:59.023458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:59.023368    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:59.086193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:59.086162    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" path="/var/lib/kubelet/pods/f02f66f5-b9f5-4cf4-bb36-dbf981efb16f/volumes"
Nov 03 13:42:59.156901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:59.156847    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:59.453854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:59.453825    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:42:59.608811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:42:59.608779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:00.073824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:00.073788    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"]
Nov 03 13:43:00.079481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:00.079458    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b7b145c-9ngj8"]
Nov 03 13:43:00.460015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:00.459954    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:01.085779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:01.085747    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0298474e-45a1-4b2b-86b1-cb1fbebb591e" path="/var/lib/kubelet/pods/0298474e-45a1-4b2b-86b1-cb1fbebb591e/volumes"
Nov 03 13:43:01.241302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:01.241266    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:01.947293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:01.947255    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:02.191649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:02.191621    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:02.686323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:02.686293    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:03.002753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:03.002709    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:03.729959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:03.729915    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:03.897557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:03.897516    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk" podUID="479c153d-e665-4083-8848-94bf3fa8fe31" containerName="payload-processing" containerID="cri-o://b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf" gracePeriod=25
Nov 03 13:43:03.907091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:03.907063    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd" podUID="c7a4b363-0733-47bc-a0cb-69e56524d691" containerName="payload-pre-processing" containerID="cri-o://258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95" gracePeriod=25
Nov 03 13:43:04.398437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:04.398400    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:05.113497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:05.113467    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:05.728641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:05.728608    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:06.333152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:06.333036    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:07.007326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:07.003715    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:07.602257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:07.602164    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:08.423863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:08.423827    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:08.575763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:08.575732    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:08.757961 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:08.757933    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:09.059273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:09.059178    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:09.376067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:09.375987    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"]
Nov 03 13:43:09.376266 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:09.376231    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy" containerID="cri-o://86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2" gracePeriod=30
Nov 03 13:43:09.380751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:09.380724    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:09.555112 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:09.555082    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:10.175786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:10.175752    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:10.326685 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:10.326658    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:10.824630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:10.824598    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:10.889102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:10.889059    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk" podUID="479c153d-e665-4083-8848-94bf3fa8fe31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:10.889269 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:10.889064    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd" podUID="c7a4b363-0733-47bc-a0cb-69e56524d691" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:11.556048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:11.556015    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:12.330284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:12.330206    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:13.029625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:13.029591    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:13.272553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:13.272523    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:13.434072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:13.433991    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:13.758200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:13.758173    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:14.037647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.037571    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"]
Nov 03 13:43:14.049568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.049540    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51afdae-658hl"]
Nov 03 13:43:14.532263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.532155    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:14.689384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.689363    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:43:14.784721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.784690    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-envoy\") pod \"11b73f05-a195-4207-974f-a30c46e31ac1\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") "
Nov 03 13:43:14.784721 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.784730    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11b73f05-a195-4207-974f-a30c46e31ac1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b73f05-a195-4207-974f-a30c46e31ac1-istio-podinfo\") pod \"11b73f05-a195-4207-974f-a30c46e31ac1\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") "
Nov 03 13:43:14.784991 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.784779    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-istio-token\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-istio-token\") pod \"11b73f05-a195-4207-974f-a30c46e31ac1\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") "
Nov 03 13:43:14.784991 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.784798    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-certs\") pod \"11b73f05-a195-4207-974f-a30c46e31ac1\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") "
Nov 03 13:43:14.784991 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.784962    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b73f05-a195-4207-974f-a30c46e31ac1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b73f05-a195-4207-974f-a30c46e31ac1-istiod-ca-cert\") pod \"11b73f05-a195-4207-974f-a30c46e31ac1\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") "
Nov 03 13:43:14.785166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785005    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-kube-api-access-4fcr9\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-kube-api-access-4fcr9\") pod \"11b73f05-a195-4207-974f-a30c46e31ac1\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") "
Nov 03 13:43:14.785166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785044    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-credential-socket\") pod \"11b73f05-a195-4207-974f-a30c46e31ac1\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") "
Nov 03 13:43:14.785166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785122    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-data\") pod \"11b73f05-a195-4207-974f-a30c46e31ac1\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") "
Nov 03 13:43:14.785377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785191    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-socket\") pod \"11b73f05-a195-4207-974f-a30c46e31ac1\" (UID: \"11b73f05-a195-4207-974f-a30c46e31ac1\") "
Nov 03 13:43:14.785501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785479    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-credential-socket" pod "11b73f05-a195-4207-974f-a30c46e31ac1" (UID: "11b73f05-a195-4207-974f-a30c46e31ac1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:14.785575 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785523    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b73f05-a195-4207-974f-a30c46e31ac1-istiod-ca-cert" pod "11b73f05-a195-4207-974f-a30c46e31ac1" (UID: "11b73f05-a195-4207-974f-a30c46e31ac1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:14.785929 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785906    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b73f05-a195-4207-974f-a30c46e31ac1-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.786121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.786099    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.786121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785918    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-socket" pod "11b73f05-a195-4207-974f-a30c46e31ac1" (UID: "11b73f05-a195-4207-974f-a30c46e31ac1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:14.786337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785951    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-certs" pod "11b73f05-a195-4207-974f-a30c46e31ac1" (UID: "11b73f05-a195-4207-974f-a30c46e31ac1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:14.786337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.785965    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-data" pod "11b73f05-a195-4207-974f-a30c46e31ac1" (UID: "11b73f05-a195-4207-974f-a30c46e31ac1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:14.787591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.787552    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11b73f05-a195-4207-974f-a30c46e31ac1-istio-podinfo" pod "11b73f05-a195-4207-974f-a30c46e31ac1" (UID: "11b73f05-a195-4207-974f-a30c46e31ac1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:43:14.787859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.787839    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-istio-token" pod "11b73f05-a195-4207-974f-a30c46e31ac1" (UID: "11b73f05-a195-4207-974f-a30c46e31ac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:14.788306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.788283    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-envoy" pod "11b73f05-a195-4207-974f-a30c46e31ac1" (UID: "11b73f05-a195-4207-974f-a30c46e31ac1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:14.789169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.789142    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-kube-api-access-4fcr9" pod "11b73f05-a195-4207-974f-a30c46e31ac1" (UID: "11b73f05-a195-4207-974f-a30c46e31ac1"). InnerVolumeSpecName "kube-api-access-4fcr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:14.887801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.887773    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fcr9\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-kube-api-access-4fcr9\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.887801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.887798    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.887801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.887807    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.888046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.887817    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.888046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.887830    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b73f05-a195-4207-974f-a30c46e31ac1-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.888046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.887840    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b73f05-a195-4207-974f-a30c46e31ac1-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.888046 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.887850    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b73f05-a195-4207-974f-a30c46e31ac1-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:14.902890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:14.902861    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:15.068314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.068140    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:15.140234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.136309    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b8cc53-7ecc-4292-82e3-6c4eb5226009" path="/var/lib/kubelet/pods/93b8cc53-7ecc-4292-82e3-6c4eb5226009/volumes"
Nov 03 13:43:15.300916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.300883    5420 generic.go:356] "Generic (PLEG): container finished" podID="11b73f05-a195-4207-974f-a30c46e31ac1" containerID="86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2" exitCode=0
Nov 03 13:43:15.301081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.300938    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"
Nov 03 13:43:15.301081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.300961    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" event={"ID":"11b73f05-a195-4207-974f-a30c46e31ac1","Type":"ContainerDied","Data":"86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2"}
Nov 03 13:43:15.301081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.301011    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g" event={"ID":"11b73f05-a195-4207-974f-a30c46e31ac1","Type":"ContainerDied","Data":"fb2d10c4af7f64f144439f3153104b1b8a36263ec335dcfbf5caa796cc8d69d5"}
Nov 03 13:43:15.301081 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.301037    5420 scope.go:122] "RemoveContainer" containerID="86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2"
Nov 03 13:43:15.312130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.312106    5420 scope.go:122] "RemoveContainer" containerID="eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243"
Nov 03 13:43:15.325898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.325873    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"]
Nov 03 13:43:15.326013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.325986    5420 scope.go:122] "RemoveContainer" containerID="86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2"
Nov 03 13:43:15.326357 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:15.326330    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2\": container with ID starting with 86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2 not found: ID does not exist" containerID="86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2"
Nov 03 13:43:15.326437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.326367    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2"} err="failed to get container status \"86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2\": rpc error: code = NotFound desc = could not find container \"86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2\": container with ID starting with 86f30c0baa3440f3e9ed8676178d6e01605f909edbe91cb60f66648c6dce87c2 not found: ID does not exist"
Nov 03 13:43:15.326437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.326391    5420 scope.go:122] "RemoveContainer" containerID="eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243"
Nov 03 13:43:15.326717 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:15.326693    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243\": container with ID starting with eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243 not found: ID does not exist" containerID="eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243"
Nov 03 13:43:15.326775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.326728    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243"} err="failed to get container status \"eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243\": rpc error: code = NotFound desc = could not find container \"eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243\": container with ID starting with eb28db65cc76f57bb4c718a9f8a1c2e95865e4517b20eb9376e0d6a336f69243 not found: ID does not exist"
Nov 03 13:43:15.330836 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.330815    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-7889g"]
Nov 03 13:43:15.536792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.536760    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:15.708233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:15.704356    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:16.506940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:16.506913    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:16.590850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:16.590817    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"]
Nov 03 13:43:16.614885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:16.614853    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"]
Nov 03 13:43:16.720752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:16.720724    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:16.865032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:16.864940    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:17.085538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:17.085509    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" path="/var/lib/kubelet/pods/11b73f05-a195-4207-974f-a30c46e31ac1/volumes"
Nov 03 13:43:17.247933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:17.247901    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:17.394338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:17.394304    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:18.022715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:18.022682    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:18.663666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:18.663638    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:19.252294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:19.252260    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:19.577980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:19.577896    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:19.719982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:19.719952    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:20.317920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:20.317889    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:20.606507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:20.606431    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:43:20.889042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:20.888970    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk" podUID="479c153d-e665-4083-8848-94bf3fa8fe31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:20.889042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:20.889019    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd" podUID="c7a4b363-0733-47bc-a0cb-69e56524d691" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:21.105495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:21.105466    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:21.591359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:21.591262    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh" podUID="addc9970-fe00-41b6-af12-32f604c2cd74" containerName="payload-processing" containerID="cri-o://f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6" gracePeriod=25
Nov 03 13:43:21.615171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:21.615130    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j" podUID="40e831ec-f202-437e-a628-be118023bde3" containerName="payload-pre-processing" containerID="cri-o://b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c" gracePeriod=25
Nov 03 13:43:21.786847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:21.786817    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:22.395777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:22.395739    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:22.698473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:22.698396    5420 scope.go:122] "RemoveContainer" containerID="72554307c7c5274691c8498ec6aa33dc60f004cd9553c55fd86dbf09bf32c929"
Nov 03 13:43:22.710943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:22.710921    5420 scope.go:122] "RemoveContainer" containerID="81617a2888968ca00f801d56d6a8701d7ca2dc4c694d73f3bb3f8070fe40b70f"
Nov 03 13:43:23.036962 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:23.036930    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:23.727352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:23.727311    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:23.889476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:23.889439    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:24.035643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.035612    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"]
Nov 03 13:43:24.039737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.039710    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ecc2ff-8k9rb"]
Nov 03 13:43:24.171820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.171795    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:43:24.234572 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.234547    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:43:24.268671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.268600    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479c153d-e665-4083-8848-94bf3fa8fe31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c153d-e665-4083-8848-94bf3fa8fe31-plugins-config-volume\") pod \"479c153d-e665-4083-8848-94bf3fa8fe31\" (UID: \"479c153d-e665-4083-8848-94bf3fa8fe31\") "
Nov 03 13:43:24.268671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.268642    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479c153d-e665-4083-8848-94bf3fa8fe31-kube-api-access-lg2t5\" (UniqueName: \"kubernetes.io/projected/479c153d-e665-4083-8848-94bf3fa8fe31-kube-api-access-lg2t5\") pod \"479c153d-e665-4083-8848-94bf3fa8fe31\" (UID: \"479c153d-e665-4083-8848-94bf3fa8fe31\") "
Nov 03 13:43:24.268857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.268668    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a4b363-0733-47bc-a0cb-69e56524d691-kube-api-access-bdwmh\" (UniqueName: \"kubernetes.io/projected/c7a4b363-0733-47bc-a0cb-69e56524d691-kube-api-access-bdwmh\") pod \"c7a4b363-0733-47bc-a0cb-69e56524d691\" (UID: \"c7a4b363-0733-47bc-a0cb-69e56524d691\") "
Nov 03 13:43:24.268857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.268814    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a4b363-0733-47bc-a0cb-69e56524d691-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a4b363-0733-47bc-a0cb-69e56524d691-plugins-config-volume\") pod \"c7a4b363-0733-47bc-a0cb-69e56524d691\" (UID: \"c7a4b363-0733-47bc-a0cb-69e56524d691\") "
Nov 03 13:43:24.269101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.269071    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479c153d-e665-4083-8848-94bf3fa8fe31-plugins-config-volume" pod "479c153d-e665-4083-8848-94bf3fa8fe31" (UID: "479c153d-e665-4083-8848-94bf3fa8fe31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:24.269273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.269248    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a4b363-0733-47bc-a0cb-69e56524d691-plugins-config-volume" pod "c7a4b363-0733-47bc-a0cb-69e56524d691" (UID: "c7a4b363-0733-47bc-a0cb-69e56524d691"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:24.269457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.269436    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a4b363-0733-47bc-a0cb-69e56524d691-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:24.269513 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.269464    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479c153d-e665-4083-8848-94bf3fa8fe31-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:24.271011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.270989    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479c153d-e665-4083-8848-94bf3fa8fe31-kube-api-access-lg2t5" pod "479c153d-e665-4083-8848-94bf3fa8fe31" (UID: "479c153d-e665-4083-8848-94bf3fa8fe31"). InnerVolumeSpecName "kube-api-access-lg2t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:24.271954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.271928    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a4b363-0733-47bc-a0cb-69e56524d691-kube-api-access-bdwmh" pod "c7a4b363-0733-47bc-a0cb-69e56524d691" (UID: "c7a4b363-0733-47bc-a0cb-69e56524d691"). InnerVolumeSpecName "kube-api-access-bdwmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:24.370993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.370960    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg2t5\" (UniqueName: \"kubernetes.io/projected/479c153d-e665-4083-8848-94bf3fa8fe31-kube-api-access-lg2t5\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:24.370993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.370993    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwmh\" (UniqueName: \"kubernetes.io/projected/c7a4b363-0733-47bc-a0cb-69e56524d691-kube-api-access-bdwmh\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:24.407601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.407573    5420 generic.go:356] "Generic (PLEG): container finished" podID="c7a4b363-0733-47bc-a0cb-69e56524d691" containerID="258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95" exitCode=0
Nov 03 13:43:24.407768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.407620    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"
Nov 03 13:43:24.407768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.407659    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd" event={"ID":"c7a4b363-0733-47bc-a0cb-69e56524d691","Type":"ContainerDied","Data":"258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95"}
Nov 03 13:43:24.407768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.407714    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd" event={"ID":"c7a4b363-0733-47bc-a0cb-69e56524d691","Type":"ContainerDied","Data":"76b5814d0e3973b2435c9c6b4973243c914598bd52933b5ca8ef45dc1314b53e"}
Nov 03 13:43:24.407768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.407744    5420 scope.go:122] "RemoveContainer" containerID="258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95"
Nov 03 13:43:24.409250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.409195    5420 generic.go:356] "Generic (PLEG): container finished" podID="479c153d-e665-4083-8848-94bf3fa8fe31" containerID="b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf" exitCode=0
Nov 03 13:43:24.409250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.409241    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk" event={"ID":"479c153d-e665-4083-8848-94bf3fa8fe31","Type":"ContainerDied","Data":"b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf"}
Nov 03 13:43:24.409450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.409294    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk" event={"ID":"479c153d-e665-4083-8848-94bf3fa8fe31","Type":"ContainerDied","Data":"2607ee26ce5109dde2456edb9ddecbf6e026294a279c38682393831991f8517b"}
Nov 03 13:43:24.409450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.409248    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"
Nov 03 13:43:24.435445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.435416    5420 scope.go:122] "RemoveContainer" containerID="258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95"
Nov 03 13:43:24.435929 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:24.435904    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95\": container with ID starting with 258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95 not found: ID does not exist" containerID="258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95"
Nov 03 13:43:24.436053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.435939    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95"} err="failed to get container status \"258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95\": rpc error: code = NotFound desc = could not find container \"258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95\": container with ID starting with 258ed3d95d7088769709ecc58fbc2d00812822800f612ab0964becb9e13d7a95 not found: ID does not exist"
Nov 03 13:43:24.436053 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.435962    5420 scope.go:122] "RemoveContainer" containerID="b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf"
Nov 03 13:43:24.446252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.446200    5420 scope.go:122] "RemoveContainer" containerID="b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf"
Nov 03 13:43:24.446589 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:24.446563    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf\": container with ID starting with b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf not found: ID does not exist" containerID="b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf"
Nov 03 13:43:24.446690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.446596    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf"} err="failed to get container status \"b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf\": rpc error: code = NotFound desc = could not find container \"b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf\": container with ID starting with b3ee062a4f261b635a44554851c7f9d3e7c23ce8ccee8097659fef27242b4bbf not found: ID does not exist"
Nov 03 13:43:24.454579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.454554    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"]
Nov 03 13:43:24.463754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.463732    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef1693-764b9bdc59-qmhqd"]
Nov 03 13:43:24.481147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.481123    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"]
Nov 03 13:43:24.484523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.484502    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef1693-54fb5894cf-ljzhk"]
Nov 03 13:43:24.739147 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:24.739110    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:43:24.807730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.807701    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"]
Nov 03 13:43:24.808285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808270    5420 cpu_manager.go:414] "containerMap: removing container" podUID="479c153d-e665-4083-8848-94bf3fa8fe31" containerName="payload-processing"
Nov 03 13:43:24.808285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808285    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479c153d-e665-4083-8848-94bf3fa8fe31" containerName="payload-processing"
Nov 03 13:43:24.808391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808313    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy"
Nov 03 13:43:24.808391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808318    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy"
Nov 03 13:43:24.808391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808335    5420 cpu_manager.go:414] "containerMap: removing container" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy"
Nov 03 13:43:24.808391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808340    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy"
Nov 03 13:43:24.808391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808351    5420 cpu_manager.go:414] "containerMap: removing container" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy"
Nov 03 13:43:24.808391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808356    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy"
Nov 03 13:43:24.808391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808363    5420 cpu_manager.go:414] "containerMap: removing container" podUID="c7a4b363-0733-47bc-a0cb-69e56524d691" containerName="payload-pre-processing"
Nov 03 13:43:24.808391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808368    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a4b363-0733-47bc-a0cb-69e56524d691" containerName="payload-pre-processing"
Nov 03 13:43:24.808681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808443    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy"
Nov 03 13:43:24.808681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808453    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a4b363-0733-47bc-a0cb-69e56524d691" containerName="payload-pre-processing"
Nov 03 13:43:24.808681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808461    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="479c153d-e665-4083-8848-94bf3fa8fe31" containerName="payload-processing"
Nov 03 13:43:24.808681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.808471    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02f66f5-b9f5-4cf4-bb36-dbf981efb16f" containerName="istio-proxy"
Nov 03 13:43:24.813959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.813937    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:24.823107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.821467    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:43:24.823107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.821808    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"]
Nov 03 13:43:24.823107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.822006    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:43:24.876148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.876118    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73fd1243-872p9\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:24.876380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.876162    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgcl5\" (UniqueName: \"kubernetes.io/projected/baba52f8-142f-498f-a2c4-ed267f98a98f-kube-api-access-xgcl5\") pod \"maas-api-revoke-sub-73fd1243-872p9\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:24.876380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.876335    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73fd1243-872p9\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:24.882763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.882740    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:43:24.883074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.883058    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:43:24.978277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.978205    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73fd1243-872p9\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:24.978495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.978313    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgcl5\" (UniqueName: \"kubernetes.io/projected/baba52f8-142f-498f-a2c4-ed267f98a98f-kube-api-access-xgcl5\") pod \"maas-api-revoke-sub-73fd1243-872p9\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:24.978495 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.978400    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73fd1243-872p9\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:24.979149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.979124    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73fd1243-872p9\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:24.984914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.983009    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73fd1243-872p9\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:24.997313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:24.996532    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgcl5\" (UniqueName: \"kubernetes.io/projected/baba52f8-142f-498f-a2c4-ed267f98a98f-kube-api-access-xgcl5\") pod \"maas-api-revoke-sub-73fd1243-872p9\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:25.084967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.084937    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479c153d-e665-4083-8848-94bf3fa8fe31" path="/var/lib/kubelet/pods/479c153d-e665-4083-8848-94bf3fa8fe31/volumes"
Nov 03 13:43:25.085403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.085389    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b1ad37-709c-403d-8a4b-27712f6eb522" path="/var/lib/kubelet/pods/b6b1ad37-709c-403d-8a4b-27712f6eb522/volumes"
Nov 03 13:43:25.085757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.085745    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a4b363-0733-47bc-a0cb-69e56524d691" path="/var/lib/kubelet/pods/c7a4b363-0733-47bc-a0cb-69e56524d691/volumes"
Nov 03 13:43:25.130960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.130939    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:25.217202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.217174    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:25.410171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.410079    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:25.420130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.420098    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"]
Nov 03 13:43:25.421162 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:43:25.421138    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaba52f8_142f_498f_a2c4_ed267f98a98f.slice/crio-2d303b3f09c9b05ae525b5348b1c611b2b4a8bf9b5ea4fe71397b538f57aa90f WatchSource:0}: Error finding container 2d303b3f09c9b05ae525b5348b1c611b2b4a8bf9b5ea4fe71397b538f57aa90f: Status 404 returned error can't find the container with id 2d303b3f09c9b05ae525b5348b1c611b2b4a8bf9b5ea4fe71397b538f57aa90f
Nov 03 13:43:25.656255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.656197    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:25.733664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.733641    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"]
Nov 03 13:43:25.734379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.734364    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b73f05-a195-4207-974f-a30c46e31ac1" containerName="istio-proxy"
Nov 03 13:43:25.737816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.737798    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:25.749457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.749435    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"]
Nov 03 13:43:25.788320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.788288    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98vv\" (UniqueName: \"kubernetes.io/projected/afdafece-fcbf-473a-b5e3-818b4761abcb-kube-api-access-b98vv\") pod \"maas-api-revoke-sub-3829d1a7-568dd\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:25.788442 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.788352    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3829d1a7-568dd\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:25.788442 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.788379    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3829d1a7-568dd\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:25.890154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.890124    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98vv\" (UniqueName: \"kubernetes.io/projected/afdafece-fcbf-473a-b5e3-818b4761abcb-kube-api-access-b98vv\") pod \"maas-api-revoke-sub-3829d1a7-568dd\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:25.890361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.890173    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3829d1a7-568dd\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:25.890361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.890203    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3829d1a7-568dd\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:25.890831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.890809    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3829d1a7-568dd\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:25.892630 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.892604    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3829d1a7-568dd\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:25.901782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:25.901758    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98vv\" (UniqueName: \"kubernetes.io/projected/afdafece-fcbf-473a-b5e3-818b4761abcb-kube-api-access-b98vv\") pod \"maas-api-revoke-sub-3829d1a7-568dd\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:26.053782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:26.053713    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:26.347056 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:26.347031    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"]
Nov 03 13:43:26.384568 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:43:26.384541    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdafece_fcbf_473a_b5e3_818b4761abcb.slice/crio-63e5505671a772b9f4856959f57773626b40101c25ba8a0e6de45036eb478e48 WatchSource:0}: Error finding container 63e5505671a772b9f4856959f57773626b40101c25ba8a0e6de45036eb478e48: Status 404 returned error can't find the container with id 63e5505671a772b9f4856959f57773626b40101c25ba8a0e6de45036eb478e48
Nov 03 13:43:26.438263 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:26.438230    5420 generic.go:356] "Generic (PLEG): container finished" podID="baba52f8-142f-498f-a2c4-ed267f98a98f" containerID="21f021823c679c6b6ae81d7856daf1d80aaf2b483eeb429945cb91ed7d236cbb" exitCode=0
Nov 03 13:43:26.438393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:26.438321    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9" event={"ID":"baba52f8-142f-498f-a2c4-ed267f98a98f","Type":"ContainerDied","Data":"21f021823c679c6b6ae81d7856daf1d80aaf2b483eeb429945cb91ed7d236cbb"}
Nov 03 13:43:26.438393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:26.438390    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9" event={"ID":"baba52f8-142f-498f-a2c4-ed267f98a98f","Type":"ContainerStarted","Data":"2d303b3f09c9b05ae525b5348b1c611b2b4a8bf9b5ea4fe71397b538f57aa90f"}
Nov 03 13:43:26.439612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:26.439592    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd" event={"ID":"afdafece-fcbf-473a-b5e3-818b4761abcb","Type":"ContainerStarted","Data":"63e5505671a772b9f4856959f57773626b40101c25ba8a0e6de45036eb478e48"}
Nov 03 13:43:26.498821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:26.498498    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:26.949784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:26.949717    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"]
Nov 03 13:43:26.950313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:26.950024    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerName="istio-proxy" containerID="cri-o://f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571" gracePeriod=30
Nov 03 13:43:27.233814 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.233779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:27.448936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.448901    5420 generic.go:356] "Generic (PLEG): container finished" podID="afdafece-fcbf-473a-b5e3-818b4761abcb" containerID="9f1005d6e6ec0fa02b7a90dd452cc74f1fe7f968bf1f2681cd0d8a083fed9500" exitCode=0
Nov 03 13:43:27.449074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.448995    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd" event={"ID":"afdafece-fcbf-473a-b5e3-818b4761abcb","Type":"ContainerDied","Data":"9f1005d6e6ec0fa02b7a90dd452cc74f1fe7f968bf1f2681cd0d8a083fed9500"}
Nov 03 13:43:27.499857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.499779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:27.519738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.519700    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j" podUID="40e831ec-f202-437e-a628-be118023bde3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:27.519738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.519724    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh" podUID="addc9970-fe00-41b6-af12-32f604c2cd74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:27.592559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.592536    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:27.611857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.611830    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-xks-ca\") pod \"baba52f8-142f-498f-a2c4-ed267f98a98f\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") "
Nov 03 13:43:27.612015 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.612003    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baba52f8-142f-498f-a2c4-ed267f98a98f-kube-api-access-xgcl5\" (UniqueName: \"kubernetes.io/projected/baba52f8-142f-498f-a2c4-ed267f98a98f-kube-api-access-xgcl5\") pod \"baba52f8-142f-498f-a2c4-ed267f98a98f\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") "
Nov 03 13:43:27.612093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.612074    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-openshift-ca\") pod \"baba52f8-142f-498f-a2c4-ed267f98a98f\" (UID: \"baba52f8-142f-498f-a2c4-ed267f98a98f\") "
Nov 03 13:43:27.612455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.612426    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-openshift-ca" pod "baba52f8-142f-498f-a2c4-ed267f98a98f" (UID: "baba52f8-142f-498f-a2c4-ed267f98a98f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:27.613898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.613878    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-xks-ca" pod "baba52f8-142f-498f-a2c4-ed267f98a98f" (UID: "baba52f8-142f-498f-a2c4-ed267f98a98f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:27.614067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.614045    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baba52f8-142f-498f-a2c4-ed267f98a98f-kube-api-access-xgcl5" pod "baba52f8-142f-498f-a2c4-ed267f98a98f" (UID: "baba52f8-142f-498f-a2c4-ed267f98a98f"). InnerVolumeSpecName "kube-api-access-xgcl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:27.713895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.713860    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:27.713895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.713891    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baba52f8-142f-498f-a2c4-ed267f98a98f-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:27.713895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:27.713903    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgcl5\" (UniqueName: \"kubernetes.io/projected/baba52f8-142f-498f-a2c4-ed267f98a98f-kube-api-access-xgcl5\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:28.242796 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.242763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:28.457821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.457791    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"
Nov 03 13:43:28.457990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.457845    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9" event={"ID":"baba52f8-142f-498f-a2c4-ed267f98a98f","Type":"ContainerDied","Data":"2d303b3f09c9b05ae525b5348b1c611b2b4a8bf9b5ea4fe71397b538f57aa90f"}
Nov 03 13:43:28.457990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.457893    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d303b3f09c9b05ae525b5348b1c611b2b4a8bf9b5ea4fe71397b538f57aa90f"
Nov 03 13:43:28.611611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.611590    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:28.725090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.725058    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdafece-fcbf-473a-b5e3-818b4761abcb-kube-api-access-b98vv\" (UniqueName: \"kubernetes.io/projected/afdafece-fcbf-473a-b5e3-818b4761abcb-kube-api-access-b98vv\") pod \"afdafece-fcbf-473a-b5e3-818b4761abcb\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") "
Nov 03 13:43:28.725311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.725159    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-xks-ca\") pod \"afdafece-fcbf-473a-b5e3-818b4761abcb\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") "
Nov 03 13:43:28.725311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.725203    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-openshift-ca\") pod \"afdafece-fcbf-473a-b5e3-818b4761abcb\" (UID: \"afdafece-fcbf-473a-b5e3-818b4761abcb\") "
Nov 03 13:43:28.725602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.725578    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-openshift-ca" pod "afdafece-fcbf-473a-b5e3-818b4761abcb" (UID: "afdafece-fcbf-473a-b5e3-818b4761abcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:28.725700 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.725681    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:28.727356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.727328    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdafece-fcbf-473a-b5e3-818b4761abcb-kube-api-access-b98vv" pod "afdafece-fcbf-473a-b5e3-818b4761abcb" (UID: "afdafece-fcbf-473a-b5e3-818b4761abcb"). InnerVolumeSpecName "kube-api-access-b98vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:28.727356 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.727340    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-xks-ca" pod "afdafece-fcbf-473a-b5e3-818b4761abcb" (UID: "afdafece-fcbf-473a-b5e3-818b4761abcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:28.827206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.827130    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b98vv\" (UniqueName: \"kubernetes.io/projected/afdafece-fcbf-473a-b5e3-818b4761abcb-kube-api-access-b98vv\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:28.827206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.827157    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdafece-fcbf-473a-b5e3-818b4761abcb-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:28.934038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:28.934002    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:29.467133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:29.467099    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"
Nov 03 13:43:29.467671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:29.467154    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd" event={"ID":"afdafece-fcbf-473a-b5e3-818b4761abcb","Type":"ContainerDied","Data":"63e5505671a772b9f4856959f57773626b40101c25ba8a0e6de45036eb478e48"}
Nov 03 13:43:29.467671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:29.467200    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e5505671a772b9f4856959f57773626b40101c25ba8a0e6de45036eb478e48"
Nov 03 13:43:29.748106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:29.748020    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:30.285515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:30.285473    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"]
Nov 03 13:43:30.290853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:30.290830    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"]
Nov 03 13:43:30.297614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:30.297592    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"]
Nov 03 13:43:30.309648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:30.309626    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"]
Nov 03 13:43:30.536892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:30.536814    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:30.711118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:30.711091    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:31.093809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:31.093772    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:31.554842 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:31.554814    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:32.214466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.214413    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:43:32.283599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.283562    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-data\") pod \"d85bb78e-5822-49f1-b38c-de584bd60c6b\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") "
Nov 03 13:43:32.283802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.283661    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-token\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-token\") pod \"d85bb78e-5822-49f1-b38c-de584bd60c6b\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") "
Nov 03 13:43:32.283802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.283692    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-podinfo\") pod \"d85bb78e-5822-49f1-b38c-de584bd60c6b\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") "
Nov 03 13:43:32.283802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.283731    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-kube-api-access-lznvd\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-kube-api-access-lznvd\") pod \"d85bb78e-5822-49f1-b38c-de584bd60c6b\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") "
Nov 03 13:43:32.284000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.283850    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-data" pod "d85bb78e-5822-49f1-b38c-de584bd60c6b" (UID: "d85bb78e-5822-49f1-b38c-de584bd60c6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:32.284000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.283921    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-envoy\") pod \"d85bb78e-5822-49f1-b38c-de584bd60c6b\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") "
Nov 03 13:43:32.284000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.283985    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85bb78e-5822-49f1-b38c-de584bd60c6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85bb78e-5822-49f1-b38c-de584bd60c6b-istiod-ca-cert\") pod \"d85bb78e-5822-49f1-b38c-de584bd60c6b\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") "
Nov 03 13:43:32.284182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.284060    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-socket\") pod \"d85bb78e-5822-49f1-b38c-de584bd60c6b\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") "
Nov 03 13:43:32.284182 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.284145    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-credential-socket\") pod \"d85bb78e-5822-49f1-b38c-de584bd60c6b\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") "
Nov 03 13:43:32.284372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.284282    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-certs\") pod \"d85bb78e-5822-49f1-b38c-de584bd60c6b\" (UID: \"d85bb78e-5822-49f1-b38c-de584bd60c6b\") "
Nov 03 13:43:32.284483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.284460    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-socket" pod "d85bb78e-5822-49f1-b38c-de584bd60c6b" (UID: "d85bb78e-5822-49f1-b38c-de584bd60c6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:32.284566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.284487    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85bb78e-5822-49f1-b38c-de584bd60c6b-istiod-ca-cert" pod "d85bb78e-5822-49f1-b38c-de584bd60c6b" (UID: "d85bb78e-5822-49f1-b38c-de584bd60c6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:32.284566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.284530    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-credential-socket" pod "d85bb78e-5822-49f1-b38c-de584bd60c6b" (UID: "d85bb78e-5822-49f1-b38c-de584bd60c6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:32.285132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.284784    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-certs" pod "d85bb78e-5822-49f1-b38c-de584bd60c6b" (UID: "d85bb78e-5822-49f1-b38c-de584bd60c6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:32.285132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.285036    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:32.285132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.285064    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85bb78e-5822-49f1-b38c-de584bd60c6b-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:32.285132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.285080    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:32.285132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.285094    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:32.285132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.285108    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:32.286171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.286143    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-token" pod "d85bb78e-5822-49f1-b38c-de584bd60c6b" (UID: "d85bb78e-5822-49f1-b38c-de584bd60c6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:32.286476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.286455    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-podinfo" pod "d85bb78e-5822-49f1-b38c-de584bd60c6b" (UID: "d85bb78e-5822-49f1-b38c-de584bd60c6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:43:32.286532 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.286452    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-envoy" pod "d85bb78e-5822-49f1-b38c-de584bd60c6b" (UID: "d85bb78e-5822-49f1-b38c-de584bd60c6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:32.286629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.286613    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-kube-api-access-lznvd" pod "d85bb78e-5822-49f1-b38c-de584bd60c6b" (UID: "d85bb78e-5822-49f1-b38c-de584bd60c6b"). InnerVolumeSpecName "kube-api-access-lznvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:32.306358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.306337    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:32.386077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.386045    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:32.386077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.386074    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:32.386077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.386085    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85bb78e-5822-49f1-b38c-de584bd60c6b-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:32.386355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.386095    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lznvd\" (UniqueName: \"kubernetes.io/projected/d85bb78e-5822-49f1-b38c-de584bd60c6b-kube-api-access-lznvd\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:32.489195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.489108    5420 generic.go:356] "Generic (PLEG): container finished" podID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerID="f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571" exitCode=0
Nov 03 13:43:32.489195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.489164    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"
Nov 03 13:43:32.489460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.489197    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" event={"ID":"d85bb78e-5822-49f1-b38c-de584bd60c6b","Type":"ContainerDied","Data":"f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571"}
Nov 03 13:43:32.489460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.489278    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz" event={"ID":"d85bb78e-5822-49f1-b38c-de584bd60c6b","Type":"ContainerDied","Data":"3d26daa26f01f15030bc7487426f8c9bb7ed01b1fb53693f83a43d41086a5706"}
Nov 03 13:43:32.489460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.489310    5420 scope.go:122] "RemoveContainer" containerID="f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571"
Nov 03 13:43:32.502446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.502368    5420 scope.go:122] "RemoveContainer" containerID="f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571"
Nov 03 13:43:32.502735 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:32.502680    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571\": container with ID starting with f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571 not found: ID does not exist" containerID="f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571"
Nov 03 13:43:32.502735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.502711    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571"} err="failed to get container status \"f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571\": rpc error: code = NotFound desc = could not find container \"f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571\": container with ID starting with f05b71af4afc0561163b3694942328df14ad96959dec9f547c1dd22276cfd571 not found: ID does not exist"
Nov 03 13:43:32.520882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.520862    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"]
Nov 03 13:43:32.528291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:32.528271    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-xznhz"]
Nov 03 13:43:33.084626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:33.084594    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" path="/var/lib/kubelet/pods/d85bb78e-5822-49f1-b38c-de584bd60c6b/volumes"
Nov 03 13:43:33.295075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:33.295046    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:34.064871 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.064839    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:34.884022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.883953    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:34.997150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.997121    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"]
Nov 03 13:43:34.997872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.997851    5420 cpu_manager.go:414] "containerMap: removing container" podUID="baba52f8-142f-498f-a2c4-ed267f98a98f" containerName="revoke-keys"
Nov 03 13:43:34.997970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.997874    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baba52f8-142f-498f-a2c4-ed267f98a98f" containerName="revoke-keys"
Nov 03 13:43:34.997970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.997919    5420 cpu_manager.go:414] "containerMap: removing container" podUID="afdafece-fcbf-473a-b5e3-818b4761abcb" containerName="revoke-keys"
Nov 03 13:43:34.997970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.997928    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdafece-fcbf-473a-b5e3-818b4761abcb" containerName="revoke-keys"
Nov 03 13:43:34.997970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.997962    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerName="istio-proxy"
Nov 03 13:43:34.997970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.997970    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerName="istio-proxy"
Nov 03 13:43:34.998169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.998155    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="baba52f8-142f-498f-a2c4-ed267f98a98f" containerName="revoke-keys"
Nov 03 13:43:34.998206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.998176    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdafece-fcbf-473a-b5e3-818b4761abcb" containerName="revoke-keys"
Nov 03 13:43:34.998206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:34.998188    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85bb78e-5822-49f1-b38c-de584bd60c6b" containerName="istio-proxy"
Nov 03 13:43:35.002815 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.002797    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.008853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.008755    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:43:35.009073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.009053    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:43:35.009227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.009104    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:43:35.009227 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.009156    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:43:35.012063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.012036    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"]
Nov 03 13:43:35.036582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.036556    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"]
Nov 03 13:43:35.041435 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.041413    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8469b741-w2hl6"]
Nov 03 13:43:35.084905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.084882    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5039a0c-e0a7-4520-b051-f50048f1a663" path="/var/lib/kubelet/pods/f5039a0c-e0a7-4520-b051-f50048f1a663/volumes"
Nov 03 13:43:35.118037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.118010    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/62e57bec-211a-4e21-913e-f85c806a45bf-kube-api-access-sd9bm\") pod \"maas-api-revoke-sub-9e2bff93-mhttp\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.118136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.118089    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2bff93-mhttp\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.118248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.118205    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2bff93-mhttp\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.219246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.219206    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2bff93-mhttp\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.219391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.219330    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2bff93-mhttp\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.219437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.219409    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/62e57bec-211a-4e21-913e-f85c806a45bf-kube-api-access-sd9bm\") pod \"maas-api-revoke-sub-9e2bff93-mhttp\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.219826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.219800    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2bff93-mhttp\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.221610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.221588    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2bff93-mhttp\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.232879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.232855    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/62e57bec-211a-4e21-913e-f85c806a45bf-kube-api-access-sd9bm\") pod \"maas-api-revoke-sub-9e2bff93-mhttp\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.321380 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.321350    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:35.613416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.613386    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"]
Nov 03 13:43:35.656360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:35.656335    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:36.402525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:36.402496    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:36.519973 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:36.519932    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp" event={"ID":"62e57bec-211a-4e21-913e-f85c806a45bf","Type":"ContainerStarted","Data":"14515f7b06d9939b003abb0593e746c95d2f97ff175d4b9e1d07a6feec8c4993"}
Nov 03 13:43:36.520158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:36.519994    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp" event={"ID":"62e57bec-211a-4e21-913e-f85c806a45bf","Type":"ContainerStarted","Data":"3149fd47e9b2679466c3b243d1cbf8860f5993848018bcbc4b7cf9b089e20d63"}
Nov 03 13:43:36.543220 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:36.543170    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp" podStartSLOduration=2.54315655 podStartE2EDuration="2.54315655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:36.539151614 +0000 UTC m=+2057.616916641" watchObservedRunningTime="2026-11-03 13:43:36.54315655 +0000 UTC m=+2057.620921575"
Nov 03 13:43:37.024866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.024838    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:37.200795 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.200750    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"]
Nov 03 13:43:37.228976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.228940    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"]
Nov 03 13:43:37.356125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.356020    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:37.519580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.519545    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j" podUID="40e831ec-f202-437e-a628-be118023bde3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:37.519957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.519615    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh" podUID="addc9970-fe00-41b6-af12-32f604c2cd74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:37.522827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.522807    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:37.529707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.529687    5420 generic.go:356] "Generic (PLEG): container finished" podID="62e57bec-211a-4e21-913e-f85c806a45bf" containerID="14515f7b06d9939b003abb0593e746c95d2f97ff175d4b9e1d07a6feec8c4993" exitCode=0
Nov 03 13:43:37.529804 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.529780    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp" event={"ID":"62e57bec-211a-4e21-913e-f85c806a45bf","Type":"ContainerDied","Data":"14515f7b06d9939b003abb0593e746c95d2f97ff175d4b9e1d07a6feec8c4993"}
Nov 03 13:43:37.892903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.891897    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:37.901293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.901264    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"]
Nov 03 13:43:37.906611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.906582    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:37.917195 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:37.917172    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"]
Nov 03 13:43:38.006538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.006504    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3547802b-szkgb\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.006715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.006606    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3547802b-szkgb\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.006715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.006639    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkx8g\" (UniqueName: \"kubernetes.io/projected/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-kube-api-access-nkx8g\") pod \"maas-api-revoke-sub-3547802b-szkgb\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.107690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.107651    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3547802b-szkgb\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.107891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.107718    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3547802b-szkgb\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.107891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.107749    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkx8g\" (UniqueName: \"kubernetes.io/projected/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-kube-api-access-nkx8g\") pod \"maas-api-revoke-sub-3547802b-szkgb\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.108502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.108473    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3547802b-szkgb\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.110048 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.110027    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3547802b-szkgb\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.119145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.119120    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkx8g\" (UniqueName: \"kubernetes.io/projected/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-kube-api-access-nkx8g\") pod \"maas-api-revoke-sub-3547802b-szkgb\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.224381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.224340    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:38.547248 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.547184    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"]
Nov 03 13:43:38.573506 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:43:38.573439    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db6e2e6_07ac_4d9c_9d0b_fb7746afd36a.slice/crio-d76a61b19f485f43f2b40de822a9b2dce94c1363c81708ad26e7f5d2124a1746 WatchSource:0}: Error finding container d76a61b19f485f43f2b40de822a9b2dce94c1363c81708ad26e7f5d2124a1746: Status 404 returned error can't find the container with id d76a61b19f485f43f2b40de822a9b2dce94c1363c81708ad26e7f5d2124a1746
Nov 03 13:43:38.654450 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.654351    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:38.883665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.878416    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:38.917899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.917868    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-xks-ca\") pod \"62e57bec-211a-4e21-913e-f85c806a45bf\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") "
Nov 03 13:43:38.918076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.917913    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-openshift-ca\") pod \"62e57bec-211a-4e21-913e-f85c806a45bf\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") "
Nov 03 13:43:38.918076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.917957    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e57bec-211a-4e21-913e-f85c806a45bf-kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/62e57bec-211a-4e21-913e-f85c806a45bf-kube-api-access-sd9bm\") pod \"62e57bec-211a-4e21-913e-f85c806a45bf\" (UID: \"62e57bec-211a-4e21-913e-f85c806a45bf\") "
Nov 03 13:43:38.918292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.918266    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-openshift-ca" pod "62e57bec-211a-4e21-913e-f85c806a45bf" (UID: "62e57bec-211a-4e21-913e-f85c806a45bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:38.918502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.918481    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:38.925483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.925423    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e57bec-211a-4e21-913e-f85c806a45bf-kube-api-access-sd9bm" pod "62e57bec-211a-4e21-913e-f85c806a45bf" (UID: "62e57bec-211a-4e21-913e-f85c806a45bf"). InnerVolumeSpecName "kube-api-access-sd9bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:38.949246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:38.948320    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-xks-ca" pod "62e57bec-211a-4e21-913e-f85c806a45bf" (UID: "62e57bec-211a-4e21-913e-f85c806a45bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:39.019879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.019847    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e57bec-211a-4e21-913e-f85c806a45bf-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:39.019879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.019881    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9bm\" (UniqueName: \"kubernetes.io/projected/62e57bec-211a-4e21-913e-f85c806a45bf-kube-api-access-sd9bm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:39.108817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.108791    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:39.555799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.555770    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"
Nov 03 13:43:39.556330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.555770    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp" event={"ID":"62e57bec-211a-4e21-913e-f85c806a45bf","Type":"ContainerDied","Data":"3149fd47e9b2679466c3b243d1cbf8860f5993848018bcbc4b7cf9b089e20d63"}
Nov 03 13:43:39.556330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.555904    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3149fd47e9b2679466c3b243d1cbf8860f5993848018bcbc4b7cf9b089e20d63"
Nov 03 13:43:39.557969 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.557941    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb" event={"ID":"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a","Type":"ContainerStarted","Data":"347a8876ee51906af9f345d3b4840edaad6033fa940f61490be21a5f8acfd72c"}
Nov 03 13:43:39.558153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.557994    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb" event={"ID":"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a","Type":"ContainerStarted","Data":"d76a61b19f485f43f2b40de822a9b2dce94c1363c81708ad26e7f5d2124a1746"}
Nov 03 13:43:39.603584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.603537    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb" podStartSLOduration=2.603526076 podStartE2EDuration="2.603526076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:39.599898139 +0000 UTC m=+2060.677663166" watchObservedRunningTime="2026-11-03 13:43:39.603526076 +0000 UTC m=+2060.681291102"
Nov 03 13:43:39.604816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:39.604794    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:40.344437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:40.344411    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:40.565792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:40.565759    5420 generic.go:356] "Generic (PLEG): container finished" podID="0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a" containerID="347a8876ee51906af9f345d3b4840edaad6033fa940f61490be21a5f8acfd72c" exitCode=0
Nov 03 13:43:40.566235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:40.565846    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb" event={"ID":"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a","Type":"ContainerDied","Data":"347a8876ee51906af9f345d3b4840edaad6033fa940f61490be21a5f8acfd72c"}
Nov 03 13:43:41.126009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.125976    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:41.751488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.751460    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:41.834223 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.823319    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:41.834223 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.825134    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:43:41.865260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.861702    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-kube-api-access-nkx8g\" (UniqueName: \"kubernetes.io/projected/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-kube-api-access-nkx8g\") pod \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") "
Nov 03 13:43:41.865260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.861829    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-xks-ca\") pod \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") "
Nov 03 13:43:41.865260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.861957    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-openshift-ca\") pod \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\" (UID: \"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a\") "
Nov 03 13:43:41.866470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.865661    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-openshift-ca" pod "0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a" (UID: "0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:41.884492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.883394    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-xks-ca" pod "0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a" (UID: "0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:41.884492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.884383    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-kube-api-access-nkx8g" pod "0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a" (UID: "0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a"). InnerVolumeSpecName "kube-api-access-nkx8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:41.937011 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.936980    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:43:41.965457 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.965427    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addc9970-fe00-41b6-af12-32f604c2cd74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addc9970-fe00-41b6-af12-32f604c2cd74-plugins-config-volume\") pod \"addc9970-fe00-41b6-af12-32f604c2cd74\" (UID: \"addc9970-fe00-41b6-af12-32f604c2cd74\") "
Nov 03 13:43:41.965642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.965625    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addc9970-fe00-41b6-af12-32f604c2cd74-kube-api-access-hfd6b\" (UniqueName: \"kubernetes.io/projected/addc9970-fe00-41b6-af12-32f604c2cd74-kube-api-access-hfd6b\") pod \"addc9970-fe00-41b6-af12-32f604c2cd74\" (UID: \"addc9970-fe00-41b6-af12-32f604c2cd74\") "
Nov 03 13:43:41.966026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.965771    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addc9970-fe00-41b6-af12-32f604c2cd74-plugins-config-volume" pod "addc9970-fe00-41b6-af12-32f604c2cd74" (UID: "addc9970-fe00-41b6-af12-32f604c2cd74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:41.966131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.966074    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:41.966131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.966095    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:41.966131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.966109    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addc9970-fe00-41b6-af12-32f604c2cd74-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:41.966131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.966121    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkx8g\" (UniqueName: \"kubernetes.io/projected/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a-kube-api-access-nkx8g\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:41.968636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.968609    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addc9970-fe00-41b6-af12-32f604c2cd74-kube-api-access-hfd6b" pod "addc9970-fe00-41b6-af12-32f604c2cd74" (UID: "addc9970-fe00-41b6-af12-32f604c2cd74"). InnerVolumeSpecName "kube-api-access-hfd6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:41.979400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:41.979383    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:43:42.062305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.062276    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"]
Nov 03 13:43:42.066736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.066706    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e831ec-f202-437e-a628-be118023bde3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e831ec-f202-437e-a628-be118023bde3-plugins-config-volume\") pod \"40e831ec-f202-437e-a628-be118023bde3\" (UID: \"40e831ec-f202-437e-a628-be118023bde3\") "
Nov 03 13:43:42.066876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.066816    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e831ec-f202-437e-a628-be118023bde3-kube-api-access-72xqj\" (UniqueName: \"kubernetes.io/projected/40e831ec-f202-437e-a628-be118023bde3-kube-api-access-72xqj\") pod \"40e831ec-f202-437e-a628-be118023bde3\" (UID: \"40e831ec-f202-437e-a628-be118023bde3\") "
Nov 03 13:43:42.067087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.067059    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e831ec-f202-437e-a628-be118023bde3-plugins-config-volume" pod "40e831ec-f202-437e-a628-be118023bde3" (UID: "40e831ec-f202-437e-a628-be118023bde3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:43:42.067466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.067442    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfd6b\" (UniqueName: \"kubernetes.io/projected/addc9970-fe00-41b6-af12-32f604c2cd74-kube-api-access-hfd6b\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:42.067548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.067473    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e831ec-f202-437e-a628-be118023bde3-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:42.068326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.068307    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c8e5fc-stf7k"]
Nov 03 13:43:42.068956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.068932    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e831ec-f202-437e-a628-be118023bde3-kube-api-access-72xqj" pod "40e831ec-f202-437e-a628-be118023bde3" (UID: "40e831ec-f202-437e-a628-be118023bde3"). InnerVolumeSpecName "kube-api-access-72xqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:42.168460 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.168390    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72xqj\" (UniqueName: \"kubernetes.io/projected/40e831ec-f202-437e-a628-be118023bde3-kube-api-access-72xqj\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:42.201802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.201768    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m" podUID="49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" containerName="payload-processing" containerID="cri-o://7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a" gracePeriod=25
Nov 03 13:43:42.230038 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.229994    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g" podUID="d9eda720-9a95-420a-bc9b-22d9c080177d" containerName="payload-pre-processing" containerID="cri-o://fe1dd7eaf7f2deed185d8207164a6ab77a7d1db544bb472a6e001aff089e1d09" gracePeriod=25
Nov 03 13:43:42.587401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.587365    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"
Nov 03 13:43:42.587796 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.587365    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb" event={"ID":"0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a","Type":"ContainerDied","Data":"d76a61b19f485f43f2b40de822a9b2dce94c1363c81708ad26e7f5d2124a1746"}
Nov 03 13:43:42.588034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.587836    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d76a61b19f485f43f2b40de822a9b2dce94c1363c81708ad26e7f5d2124a1746"
Nov 03 13:43:42.597028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.594432    5420 generic.go:356] "Generic (PLEG): container finished" podID="addc9970-fe00-41b6-af12-32f604c2cd74" containerID="f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6" exitCode=0
Nov 03 13:43:42.597028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.594483    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"
Nov 03 13:43:42.597028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.594523    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh" event={"ID":"addc9970-fe00-41b6-af12-32f604c2cd74","Type":"ContainerDied","Data":"f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6"}
Nov 03 13:43:42.597028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.595414    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh" event={"ID":"addc9970-fe00-41b6-af12-32f604c2cd74","Type":"ContainerDied","Data":"29d819a3202fb4facd0686c048073e55ab81dd70d32660d9499b6ba1d7f45c0a"}
Nov 03 13:43:42.597028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.595455    5420 scope.go:122] "RemoveContainer" containerID="f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6"
Nov 03 13:43:42.597807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.597094    5420 generic.go:356] "Generic (PLEG): container finished" podID="40e831ec-f202-437e-a628-be118023bde3" containerID="b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c" exitCode=0
Nov 03 13:43:42.597807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.597322    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j" event={"ID":"40e831ec-f202-437e-a628-be118023bde3","Type":"ContainerDied","Data":"b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c"}
Nov 03 13:43:42.597807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.597358    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j" event={"ID":"40e831ec-f202-437e-a628-be118023bde3","Type":"ContainerDied","Data":"de670f856c9064eeb10f0bef6809a5cb7e494c27b1e65927f0f839982629f0f0"}
Nov 03 13:43:42.597807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.597418    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"
Nov 03 13:43:42.619629 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.619607    5420 scope.go:122] "RemoveContainer" containerID="f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6"
Nov 03 13:43:42.620003 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:42.619978    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6\": container with ID starting with f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6 not found: ID does not exist" containerID="f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6"
Nov 03 13:43:42.620165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.620017    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6"} err="failed to get container status \"f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6\": rpc error: code = NotFound desc = could not find container \"f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6\": container with ID starting with f37598536649fdc26a66f68188fb7039c864252750f42bbbf552bd82f94365f6 not found: ID does not exist"
Nov 03 13:43:42.620165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.620040    5420 scope.go:122] "RemoveContainer" containerID="b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c"
Nov 03 13:43:42.635547 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.635526    5420 scope.go:122] "RemoveContainer" containerID="b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c"
Nov 03 13:43:42.635834 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:42.635809    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c\": container with ID starting with b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c not found: ID does not exist" containerID="b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c"
Nov 03 13:43:42.635934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.635838    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c"} err="failed to get container status \"b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c\": rpc error: code = NotFound desc = could not find container \"b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c\": container with ID starting with b2b71653efeb5d1b10804b17079e90141add1bcb10c4fef0b3a486cdd6c33c6c not found: ID does not exist"
Nov 03 13:43:42.650455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.650428    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"]
Nov 03 13:43:42.654470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.654450    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1a18c8-86d889f759-ql96j"]
Nov 03 13:43:42.674223 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.674180    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"]
Nov 03 13:43:42.680230 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:42.680197    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1a18c8-5d757dc685-8vdgh"]
Nov 03 13:43:43.102232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:43.102142    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e831ec-f202-437e-a628-be118023bde3" path="/var/lib/kubelet/pods/40e831ec-f202-437e-a628-be118023bde3/volumes"
Nov 03 13:43:43.102853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:43.102821    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78877620-c79b-47d5-b64d-23ea310be7c2" path="/var/lib/kubelet/pods/78877620-c79b-47d5-b64d-23ea310be7c2/volumes"
Nov 03 13:43:43.104106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:43.103440    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addc9970-fe00-41b6-af12-32f604c2cd74" path="/var/lib/kubelet/pods/addc9970-fe00-41b6-af12-32f604c2cd74/volumes"
Nov 03 13:43:43.860234 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:43.859635    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:44.046260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:44.045890    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:44.746926 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:44.746896    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:45.104320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.104242    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"]
Nov 03 13:43:45.111517 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.111496    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd29c96-z9hzr"]
Nov 03 13:43:45.307905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.307868    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main" containerID="cri-o://67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976" gracePeriod=45
Nov 03 13:43:45.313767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.313736    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerName="main" containerID="cri-o://0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e" gracePeriod=45
Nov 03 13:43:45.319282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.319197    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main" containerID="cri-o://097732f26d4575736cc1c57c48bec4b2e5f53f97c5e08dc328ed857b7b4f244c" gracePeriod=45
Nov 03 13:43:45.329940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.329892    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main" containerID="cri-o://2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf" gracePeriod=45
Nov 03 13:43:45.462133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.459765    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": dial tcp 10.134.0.95:8000: connect: connection refused"
Nov 03 13:43:45.630378 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.630351    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:43:45.642456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.642429    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:43:45.647174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.647155    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:43:45.651851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.651824    5420 generic.go:356] "Generic (PLEG): container finished" podID="93865294-a4e8-40fc-a2e2-d61659934afa" containerID="2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf" exitCode=0
Nov 03 13:43:45.651975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.651856    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"
Nov 03 13:43:45.651975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.651901    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" event={"ID":"93865294-a4e8-40fc-a2e2-d61659934afa","Type":"ContainerDied","Data":"2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf"}
Nov 03 13:43:45.651975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.651942    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz" event={"ID":"93865294-a4e8-40fc-a2e2-d61659934afa","Type":"ContainerDied","Data":"ceef284c80bc870980af76d212f0634ce1638544b41ceea868a13b6e48d77736"}
Nov 03 13:43:45.651975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.651964    5420 scope.go:122] "RemoveContainer" containerID="2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf"
Nov 03 13:43:45.653477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.653457    5420 generic.go:356] "Generic (PLEG): container finished" podID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerID="097732f26d4575736cc1c57c48bec4b2e5f53f97c5e08dc328ed857b7b4f244c" exitCode=0
Nov 03 13:43:45.653583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.653522    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" event={"ID":"9a8d910c-8088-4ad9-95ca-571f9af28bea","Type":"ContainerDied","Data":"097732f26d4575736cc1c57c48bec4b2e5f53f97c5e08dc328ed857b7b4f244c"}
Nov 03 13:43:45.655126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.655102    5420 generic.go:356] "Generic (PLEG): container finished" podID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerID="0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e" exitCode=0
Nov 03 13:43:45.655255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.655152    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"
Nov 03 13:43:45.655377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.655349    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" event={"ID":"6c5a7abc-d740-431c-b70d-d0e7190f8833","Type":"ContainerDied","Data":"0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e"}
Nov 03 13:43:45.655473 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.655391    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9" event={"ID":"6c5a7abc-d740-431c-b70d-d0e7190f8833","Type":"ContainerDied","Data":"745e2ed4e083f44acb004d80f457fc4985a25a6d36c8e40be39016316c186760"}
Nov 03 13:43:45.656928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.656904    5420 generic.go:356] "Generic (PLEG): container finished" podID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerID="67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976" exitCode=0
Nov 03 13:43:45.657020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.656946    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" event={"ID":"5750fc35-ee3b-4539-a8e1-ad04af8d50b6","Type":"ContainerDied","Data":"67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976"}
Nov 03 13:43:45.657020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.656969    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7" event={"ID":"5750fc35-ee3b-4539-a8e1-ad04af8d50b6","Type":"ContainerDied","Data":"50d0067b9aa542d6b276f36c3ed47a94d457a236bdecc01453a4ab8465ed8e34"}
Nov 03 13:43:45.657159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.657030    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"
Nov 03 13:43:45.659903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.659881    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:43:45.664758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.664745    5420 scope.go:122] "RemoveContainer" containerID="2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf"
Nov 03 13:43:45.665068 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:45.665041    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf\": container with ID starting with 2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf not found: ID does not exist" containerID="2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf"
Nov 03 13:43:45.665186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.665076    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf"} err="failed to get container status \"2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf\": rpc error: code = NotFound desc = could not find container \"2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf\": container with ID starting with 2417c223167ee9e2a2657845bbf2f7cf7a834ed7f3c7aeec1242b75ae75edfaf not found: ID does not exist"
Nov 03 13:43:45.665186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.665101    5420 scope.go:122] "RemoveContainer" containerID="0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e"
Nov 03 13:43:45.679244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.678983    5420 scope.go:122] "RemoveContainer" containerID="0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e"
Nov 03 13:43:45.679413 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:45.679391    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e\": container with ID starting with 0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e not found: ID does not exist" containerID="0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e"
Nov 03 13:43:45.679488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.679422    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e"} err="failed to get container status \"0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e\": rpc error: code = NotFound desc = could not find container \"0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e\": container with ID starting with 0c6cbbcc15811b34fd6d51ff72b40f69fb5ef9b746c7ce46a024fa98f73bf45e not found: ID does not exist"
Nov 03 13:43:45.679488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.679439    5420 scope.go:122] "RemoveContainer" containerID="67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976"
Nov 03 13:43:45.691090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.691067    5420 scope.go:122] "RemoveContainer" containerID="67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976"
Nov 03 13:43:45.692032 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:45.692002    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976\": container with ID starting with 67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976 not found: ID does not exist" containerID="67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976"
Nov 03 13:43:45.692160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.692043    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976"} err="failed to get container status \"67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976\": rpc error: code = NotFound desc = could not find container \"67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976\": container with ID starting with 67e719b3a9a22871798a1aca6ac9c78c462715405855536ea04a24268ba2b976 not found: ID does not exist"
Nov 03 13:43:45.744162 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744132    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tmp-dir\") pod \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") "
Nov 03 13:43:45.744327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744254    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-dshm\") pod \"9a8d910c-8088-4ad9-95ca-571f9af28bea\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") "
Nov 03 13:43:45.744327 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744301    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93865294-a4e8-40fc-a2e2-d61659934afa-kube-api-access-9hh9m\" (UniqueName: \"kubernetes.io/projected/93865294-a4e8-40fc-a2e2-d61659934afa-kube-api-access-9hh9m\") pod \"93865294-a4e8-40fc-a2e2-d61659934afa\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") "
Nov 03 13:43:45.744462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744365    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93865294-a4e8-40fc-a2e2-d61659934afa-tls-certs\" (UniqueName: \"kubernetes.io/secret/93865294-a4e8-40fc-a2e2-d61659934afa-tls-certs\") pod \"93865294-a4e8-40fc-a2e2-d61659934afa\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") "
Nov 03 13:43:45.744462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744427    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tmp-dir" pod "5750fc35-ee3b-4539-a8e1-ad04af8d50b6" (UID: "5750fc35-ee3b-4539-a8e1-ad04af8d50b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.744462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744445    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-home\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-home\") pod \"93865294-a4e8-40fc-a2e2-d61659934afa\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") "
Nov 03 13:43:45.744635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744469    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tls-certs\") pod \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") "
Nov 03 13:43:45.744635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744498    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-tmp-dir\") pod \"6c5a7abc-d740-431c-b70d-d0e7190f8833\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") "
Nov 03 13:43:45.744635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744526    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8d910c-8088-4ad9-95ca-571f9af28bea-kube-api-access-9d2fq\" (UniqueName: \"kubernetes.io/projected/9a8d910c-8088-4ad9-95ca-571f9af28bea-kube-api-access-9d2fq\") pod \"9a8d910c-8088-4ad9-95ca-571f9af28bea\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") "
Nov 03 13:43:45.744635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744574    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-home\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-home\") pod \"6c5a7abc-d740-431c-b70d-d0e7190f8833\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") "
Nov 03 13:43:45.744635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744620    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-model-cache\") pod \"93865294-a4e8-40fc-a2e2-d61659934afa\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") "
Nov 03 13:43:45.744911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744658    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8d910c-8088-4ad9-95ca-571f9af28bea-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8d910c-8088-4ad9-95ca-571f9af28bea-tls-certs\") pod \"9a8d910c-8088-4ad9-95ca-571f9af28bea\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") "
Nov 03 13:43:45.744911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744695    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-home\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-home\") pod \"9a8d910c-8088-4ad9-95ca-571f9af28bea\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") "
Nov 03 13:43:45.744911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744729    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-home\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-home\") pod \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") "
Nov 03 13:43:45.744911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744781    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-model-cache\") pod \"6c5a7abc-d740-431c-b70d-d0e7190f8833\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") "
Nov 03 13:43:45.744911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744832    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-tmp-dir\") pod \"93865294-a4e8-40fc-a2e2-d61659934afa\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") "
Nov 03 13:43:45.744911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744861    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-model-cache\") pod \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") "
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744930    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-dshm\") pod \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") "
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744937    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-tmp-dir" pod "6c5a7abc-d740-431c-b70d-d0e7190f8833" (UID: "6c5a7abc-d740-431c-b70d-d0e7190f8833"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744959    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-dshm\") pod \"93865294-a4e8-40fc-a2e2-d61659934afa\" (UID: \"93865294-a4e8-40fc-a2e2-d61659934afa\") "
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.744995    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-model-cache\") pod \"9a8d910c-8088-4ad9-95ca-571f9af28bea\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") "
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745052    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-tmp-dir\") pod \"9a8d910c-8088-4ad9-95ca-571f9af28bea\" (UID: \"9a8d910c-8088-4ad9-95ca-571f9af28bea\") "
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745135    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5a7abc-d740-431c-b70d-d0e7190f8833-kube-api-access-psl5d\" (UniqueName: \"kubernetes.io/projected/6c5a7abc-d740-431c-b70d-d0e7190f8833-kube-api-access-psl5d\") pod \"6c5a7abc-d740-431c-b70d-d0e7190f8833\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") "
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745169    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-dshm\") pod \"6c5a7abc-d740-431c-b70d-d0e7190f8833\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") "
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745203    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-kube-api-access-6nhmw\" (UniqueName: \"kubernetes.io/projected/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-kube-api-access-6nhmw\") pod \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\" (UID: \"5750fc35-ee3b-4539-a8e1-ad04af8d50b6\") "
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745205    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-model-cache" pod "6c5a7abc-d740-431c-b70d-d0e7190f8833" (UID: "6c5a7abc-d740-431c-b70d-d0e7190f8833"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.745387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745247    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5a7abc-d740-431c-b70d-d0e7190f8833-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5a7abc-d740-431c-b70d-d0e7190f8833-tls-certs\") pod \"6c5a7abc-d740-431c-b70d-d0e7190f8833\" (UID: \"6c5a7abc-d740-431c-b70d-d0e7190f8833\") "
Nov 03 13:43:45.745934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745450    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-model-cache" pod "93865294-a4e8-40fc-a2e2-d61659934afa" (UID: "93865294-a4e8-40fc-a2e2-d61659934afa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.745934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745667    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-tmp-dir" pod "93865294-a4e8-40fc-a2e2-d61659934afa" (UID: "93865294-a4e8-40fc-a2e2-d61659934afa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.745934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745735    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-home" pod "93865294-a4e8-40fc-a2e2-d61659934afa" (UID: "93865294-a4e8-40fc-a2e2-d61659934afa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.745934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745781    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-home" pod "6c5a7abc-d740-431c-b70d-d0e7190f8833" (UID: "6c5a7abc-d740-431c-b70d-d0e7190f8833"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.746166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745957    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.746166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745976    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.746166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745990    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.746166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.746003    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.746166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.746018    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.746166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.746030    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.746166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.746043    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.746166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.745987    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-home" pod "9a8d910c-8088-4ad9-95ca-571f9af28bea" (UID: "9a8d910c-8088-4ad9-95ca-571f9af28bea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.746166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.746149    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-tmp-dir" pod "9a8d910c-8088-4ad9-95ca-571f9af28bea" (UID: "9a8d910c-8088-4ad9-95ca-571f9af28bea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.746678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.746525    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-home" pod "5750fc35-ee3b-4539-a8e1-ad04af8d50b6" (UID: "5750fc35-ee3b-4539-a8e1-ad04af8d50b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.746940 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.746892    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-model-cache" pod "5750fc35-ee3b-4539-a8e1-ad04af8d50b6" (UID: "5750fc35-ee3b-4539-a8e1-ad04af8d50b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.747443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.747107    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-model-cache" pod "9a8d910c-8088-4ad9-95ca-571f9af28bea" (UID: "9a8d910c-8088-4ad9-95ca-571f9af28bea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.749047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.749020    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93865294-a4e8-40fc-a2e2-d61659934afa-tls-certs" pod "93865294-a4e8-40fc-a2e2-d61659934afa" (UID: "93865294-a4e8-40fc-a2e2-d61659934afa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:45.749475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.749449    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tls-certs" pod "5750fc35-ee3b-4539-a8e1-ad04af8d50b6" (UID: "5750fc35-ee3b-4539-a8e1-ad04af8d50b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:45.750909 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.750824    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8d910c-8088-4ad9-95ca-571f9af28bea-kube-api-access-9d2fq" pod "9a8d910c-8088-4ad9-95ca-571f9af28bea" (UID: "9a8d910c-8088-4ad9-95ca-571f9af28bea"). InnerVolumeSpecName "kube-api-access-9d2fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:45.750909 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.750872    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-dshm" pod "9a8d910c-8088-4ad9-95ca-571f9af28bea" (UID: "9a8d910c-8088-4ad9-95ca-571f9af28bea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.751087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.750930    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5a7abc-d740-431c-b70d-d0e7190f8833-kube-api-access-psl5d" pod "6c5a7abc-d740-431c-b70d-d0e7190f8833" (UID: "6c5a7abc-d740-431c-b70d-d0e7190f8833"). InnerVolumeSpecName "kube-api-access-psl5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:45.751087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.750805    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-dshm" pod "5750fc35-ee3b-4539-a8e1-ad04af8d50b6" (UID: "5750fc35-ee3b-4539-a8e1-ad04af8d50b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.751349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.751314    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-dshm" pod "93865294-a4e8-40fc-a2e2-d61659934afa" (UID: "93865294-a4e8-40fc-a2e2-d61659934afa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.751507 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.751480    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8d910c-8088-4ad9-95ca-571f9af28bea-tls-certs" pod "9a8d910c-8088-4ad9-95ca-571f9af28bea" (UID: "9a8d910c-8088-4ad9-95ca-571f9af28bea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:45.751636 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.751535    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93865294-a4e8-40fc-a2e2-d61659934afa-kube-api-access-9hh9m" pod "93865294-a4e8-40fc-a2e2-d61659934afa" (UID: "93865294-a4e8-40fc-a2e2-d61659934afa"). InnerVolumeSpecName "kube-api-access-9hh9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:45.752511 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.752478    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-kube-api-access-6nhmw" pod "5750fc35-ee3b-4539-a8e1-ad04af8d50b6" (UID: "5750fc35-ee3b-4539-a8e1-ad04af8d50b6"). InnerVolumeSpecName "kube-api-access-6nhmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:43:45.753258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.753199    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5a7abc-d740-431c-b70d-d0e7190f8833-tls-certs" pod "6c5a7abc-d740-431c-b70d-d0e7190f8833" (UID: "6c5a7abc-d740-431c-b70d-d0e7190f8833"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:43:45.753534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.753458    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-dshm" pod "6c5a7abc-d740-431c-b70d-d0e7190f8833" (UID: "6c5a7abc-d740-431c-b70d-d0e7190f8833"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:43:45.820132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.820097    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:45.847497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847460    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847489    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847502    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847514    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847528    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93865294-a4e8-40fc-a2e2-d61659934afa-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847538    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847549    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847561    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psl5d\" (UniqueName: \"kubernetes.io/projected/6c5a7abc-d740-431c-b70d-d0e7190f8833-kube-api-access-psl5d\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847575    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5a7abc-d740-431c-b70d-d0e7190f8833-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847587    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhmw\" (UniqueName: \"kubernetes.io/projected/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-kube-api-access-6nhmw\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847599    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5a7abc-d740-431c-b70d-d0e7190f8833-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847611    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8d910c-8088-4ad9-95ca-571f9af28bea-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847622    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hh9m\" (UniqueName: \"kubernetes.io/projected/93865294-a4e8-40fc-a2e2-d61659934afa-kube-api-access-9hh9m\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847634    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93865294-a4e8-40fc-a2e2-d61659934afa-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847645    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5750fc35-ee3b-4539-a8e1-ad04af8d50b6-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847657    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2fq\" (UniqueName: \"kubernetes.io/projected/9a8d910c-8088-4ad9-95ca-571f9af28bea-kube-api-access-9d2fq\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.847758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.847669    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8d910c-8088-4ad9-95ca-571f9af28bea-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:43:45.984567 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.984541    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"]
Nov 03 13:43:45.989382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:45.989360    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/simulator-w2-e440bc-kserve-7697659859-frndz"]
Nov 03 13:43:46.012592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.012565    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"]
Nov 03 13:43:46.060622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.059478    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:46.060622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.059799    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-2-w2-e440bc-kserve-85d96bdc8-l4pb7"]
Nov 03 13:43:46.079397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.079367    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"]
Nov 03 13:43:46.084344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.084311    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/distinct-w2-e440bc-kserve-679d8f7f8f-tvdh9"]
Nov 03 13:43:46.653347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.653254    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:46.667401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.667376    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"
Nov 03 13:43:46.667562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.667379    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg" event={"ID":"9a8d910c-8088-4ad9-95ca-571f9af28bea","Type":"ContainerDied","Data":"8d2d9b1832c7df8ddcb2ed96e69d366b9fc9f60043c8710f10cc88c6b5f978a4"}
Nov 03 13:43:46.667562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.667496    5420 scope.go:122] "RemoveContainer" containerID="097732f26d4575736cc1c57c48bec4b2e5f53f97c5e08dc328ed857b7b4f244c"
Nov 03 13:43:46.683481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.683451    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m" podUID="49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:46.693854 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.693829    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"]
Nov 03 13:43:46.696981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:46.696955    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/premium-w2-e440bc-kserve-67699b949d-6bwsg"]
Nov 03 13:43:47.085430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:47.085400    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" path="/var/lib/kubelet/pods/5750fc35-ee3b-4539-a8e1-ad04af8d50b6/volumes"
Nov 03 13:43:47.085850 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:47.085836    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" path="/var/lib/kubelet/pods/6c5a7abc-d740-431c-b70d-d0e7190f8833/volumes"
Nov 03 13:43:47.086254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:47.086241    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" path="/var/lib/kubelet/pods/93865294-a4e8-40fc-a2e2-d61659934afa/volumes"
Nov 03 13:43:47.086946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:47.086931    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" path="/var/lib/kubelet/pods/9a8d910c-8088-4ad9-95ca-571f9af28bea/volumes"
Nov 03 13:43:47.087373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:47.087359    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd0dba9-afd5-42af-8aac-5c691917fd4f" path="/var/lib/kubelet/pods/cfd0dba9-afd5-42af-8aac-5c691917fd4f/volumes"
Nov 03 13:43:47.556421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:47.556379    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:47.650558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:47.650529    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"]
Nov 03 13:43:47.806347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:47.806311    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:48.029753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:48.029717    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:48.268255 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:48.268192    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:48.589592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:48.589562    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:48.693728 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:48.693686    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g" podUID="d9eda720-9a95-420a-bc9b-22d9c080177d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:49.616260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:49.616186    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:50.540060 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:50.540031    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:51.252981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:51.252947    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:51.973884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:51.973845    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:52.817566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:52.817537    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:53.570427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:53.570398    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:54.374865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:54.374833    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:54.743425 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:43:54.743391    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:43:55.212426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:55.212395    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:55.766921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:55.766880    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:56.210762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:56.208292    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"]
Nov 03 13:43:56.396731 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:56.378662    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:56.617084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:56.616722    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:56.702925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:56.702841    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m" podUID="49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:56.879222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:56.879132    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:57.155669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:57.155582    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:57.830803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:57.830773    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:58.126252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126148    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"]
Nov 03 13:43:58.126694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126681    5420 cpu_manager.go:414] "containerMap: removing container" podUID="40e831ec-f202-437e-a628-be118023bde3" containerName="payload-pre-processing"
Nov 03 13:43:58.126744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126695    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e831ec-f202-437e-a628-be118023bde3" containerName="payload-pre-processing"
Nov 03 13:43:58.126744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126709    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a" containerName="revoke-keys"
Nov 03 13:43:58.126744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126714    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a" containerName="revoke-keys"
Nov 03 13:43:58.126744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126723    5420 cpu_manager.go:414] "containerMap: removing container" podUID="addc9970-fe00-41b6-af12-32f604c2cd74" containerName="payload-processing"
Nov 03 13:43:58.126744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126727    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addc9970-fe00-41b6-af12-32f604c2cd74" containerName="payload-processing"
Nov 03 13:43:58.126744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126742    5420 cpu_manager.go:414] "containerMap: removing container" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126748    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126758    5420 cpu_manager.go:414] "containerMap: removing container" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126763    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126782    5420 cpu_manager.go:414] "containerMap: removing container" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126786    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126793    5420 cpu_manager.go:414] "containerMap: removing container" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126797    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126806    5420 cpu_manager.go:414] "containerMap: removing container" podUID="62e57bec-211a-4e21-913e-f85c806a45bf" containerName="revoke-keys"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126810    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e57bec-211a-4e21-913e-f85c806a45bf" containerName="revoke-keys"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126878    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="93865294-a4e8-40fc-a2e2-d61659934afa" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126890    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="5750fc35-ee3b-4539-a8e1-ad04af8d50b6" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126897    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8d910c-8088-4ad9-95ca-571f9af28bea" containerName="main"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126902    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e57bec-211a-4e21-913e-f85c806a45bf" containerName="revoke-keys"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126910    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e831ec-f202-437e-a628-be118023bde3" containerName="payload-pre-processing"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126917    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a" containerName="revoke-keys"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126923    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="addc9970-fe00-41b6-af12-32f604c2cd74" containerName="payload-processing"
Nov 03 13:43:58.126924 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.126930    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5a7abc-d740-431c-b70d-d0e7190f8833" containerName="main"
Nov 03 13:43:58.132398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.132380    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.154873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.154848    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:43:58.154985 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.154886    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:43:58.155028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.154982    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:43:58.159205 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.159191    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:43:58.174456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.174430    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"]
Nov 03 13:43:58.244173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.244144    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3b8a3fc-t28d2\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.244350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.244195    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qnhx\" (UniqueName: \"kubernetes.io/projected/00298a04-971e-4bc0-a25f-b0bd162681f9-kube-api-access-6qnhx\") pod \"maas-api-revoke-sub-e3b8a3fc-t28d2\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.244421 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.244388    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3b8a3fc-t28d2\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.345240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.345196    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3b8a3fc-t28d2\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.345386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.345270    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnhx\" (UniqueName: \"kubernetes.io/projected/00298a04-971e-4bc0-a25f-b0bd162681f9-kube-api-access-6qnhx\") pod \"maas-api-revoke-sub-e3b8a3fc-t28d2\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.345386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.345316    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3b8a3fc-t28d2\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.345779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.345758    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3b8a3fc-t28d2\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.347581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.347558    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3b8a3fc-t28d2\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.361130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.361103    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnhx\" (UniqueName: \"kubernetes.io/projected/00298a04-971e-4bc0-a25f-b0bd162681f9-kube-api-access-6qnhx\") pod \"maas-api-revoke-sub-e3b8a3fc-t28d2\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.446711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.446646    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:43:58.693861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.693830    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g" podUID="d9eda720-9a95-420a-bc9b-22d9c080177d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:43:58.741148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.741069    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:58.743077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.743048    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"]
Nov 03 13:43:58.755151 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:43:58.755128    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00298a04_971e_4bc0_a25f_b0bd162681f9.slice/crio-0543008d1ef8e58e3fd59475179119f1cd901b0e15303bdf6260912496d7cbdc WatchSource:0}: Error finding container 0543008d1ef8e58e3fd59475179119f1cd901b0e15303bdf6260912496d7cbdc: Status 404 returned error can't find the container with id 0543008d1ef8e58e3fd59475179119f1cd901b0e15303bdf6260912496d7cbdc
Nov 03 13:43:58.822008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:58.821978    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2" event={"ID":"00298a04-971e-4bc0-a25f-b0bd162681f9","Type":"ContainerStarted","Data":"0543008d1ef8e58e3fd59475179119f1cd901b0e15303bdf6260912496d7cbdc"}
Nov 03 13:43:59.700644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:59.700616    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:59.830223 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:59.830184    5420 generic.go:356] "Generic (PLEG): container finished" podID="00298a04-971e-4bc0-a25f-b0bd162681f9" containerID="f2bdb7ff34751b538fd609e777935f152decef12bc667bd296a44f4bb7b10069" exitCode=0
Nov 03 13:43:59.830491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:43:59.830271    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2" event={"ID":"00298a04-971e-4bc0-a25f-b0bd162681f9","Type":"ContainerDied","Data":"f2bdb7ff34751b538fd609e777935f152decef12bc667bd296a44f4bb7b10069"}
Nov 03 13:44:00.481375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:00.481347    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:00.879817 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:00.879789    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:01.026115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.026086    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:44:01.085704 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.085675    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-openshift-ca\") pod \"00298a04-971e-4bc0-a25f-b0bd162681f9\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") "
Nov 03 13:44:01.085904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.085800    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00298a04-971e-4bc0-a25f-b0bd162681f9-kube-api-access-6qnhx\" (UniqueName: \"kubernetes.io/projected/00298a04-971e-4bc0-a25f-b0bd162681f9-kube-api-access-6qnhx\") pod \"00298a04-971e-4bc0-a25f-b0bd162681f9\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") "
Nov 03 13:44:01.085904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.085844    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-xks-ca\") pod \"00298a04-971e-4bc0-a25f-b0bd162681f9\" (UID: \"00298a04-971e-4bc0-a25f-b0bd162681f9\") "
Nov 03 13:44:01.086066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.086038    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-openshift-ca" pod "00298a04-971e-4bc0-a25f-b0bd162681f9" (UID: "00298a04-971e-4bc0-a25f-b0bd162681f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:01.086392 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.086369    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:01.087930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.087876    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-xks-ca" pod "00298a04-971e-4bc0-a25f-b0bd162681f9" (UID: "00298a04-971e-4bc0-a25f-b0bd162681f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:01.088059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.088038    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00298a04-971e-4bc0-a25f-b0bd162681f9-kube-api-access-6qnhx" pod "00298a04-971e-4bc0-a25f-b0bd162681f9" (UID: "00298a04-971e-4bc0-a25f-b0bd162681f9"). InnerVolumeSpecName "kube-api-access-6qnhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:01.187581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.187553    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qnhx\" (UniqueName: \"kubernetes.io/projected/00298a04-971e-4bc0-a25f-b0bd162681f9-kube-api-access-6qnhx\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:01.187581 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.187580    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00298a04-971e-4bc0-a25f-b0bd162681f9-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:01.209142 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.209111    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c" podUID="f289deff-b01c-46c1-9336-bc43d9a8023b" containerName="payload-pre-processing" containerID="cri-o://1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2" gracePeriod=25
Nov 03 13:44:01.685580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.685542    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:01.846353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.846324    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"
Nov 03 13:44:01.846526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.846363    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2" event={"ID":"00298a04-971e-4bc0-a25f-b0bd162681f9","Type":"ContainerDied","Data":"0543008d1ef8e58e3fd59475179119f1cd901b0e15303bdf6260912496d7cbdc"}
Nov 03 13:44:01.846526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.846455    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0543008d1ef8e58e3fd59475179119f1cd901b0e15303bdf6260912496d7cbdc"
Nov 03 13:44:01.907548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:01.907520    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:44:02.581489 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.580823    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:02.628984 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.628951    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:44:02.717331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.717197    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-kube-api-access-92cn2\" (UniqueName: \"kubernetes.io/projected/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-kube-api-access-92cn2\") pod \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\" (UID: \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\") "
Nov 03 13:44:02.735426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.732565    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx" podUID="e1c82ef7-9eee-408d-8d33-0fd166280fba" containerName="main" containerID="cri-o://46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944" gracePeriod=45
Nov 03 13:44:02.735426 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.734674    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-plugins-config-volume\") pod \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\" (UID: \"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80\") "
Nov 03 13:44:02.735827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.735797    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-plugins-config-volume" pod "49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" (UID: "49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:02.764394 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.764355    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-kube-api-access-92cn2" pod "49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" (UID: "49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80"). InnerVolumeSpecName "kube-api-access-92cn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:02.765605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.765573    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:02.811243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.805155    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx" podUID="e1c82ef7-9eee-408d-8d33-0fd166280fba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.238:8000/ready\": dial tcp 10.134.0.238:8000: connect: connection refused"
Nov 03 13:44:02.847294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.838193    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:02.847294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.840198    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92cn2\" (UniqueName: \"kubernetes.io/projected/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80-kube-api-access-92cn2\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:02.902243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.890612    5420 generic.go:356] "Generic (PLEG): container finished" podID="d9eda720-9a95-420a-bc9b-22d9c080177d" containerID="fe1dd7eaf7f2deed185d8207164a6ab77a7d1db544bb472a6e001aff089e1d09" exitCode=0
Nov 03 13:44:02.902243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.890726    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g" event={"ID":"d9eda720-9a95-420a-bc9b-22d9c080177d","Type":"ContainerDied","Data":"fe1dd7eaf7f2deed185d8207164a6ab77a7d1db544bb472a6e001aff089e1d09"}
Nov 03 13:44:02.902243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.892370    5420 generic.go:356] "Generic (PLEG): container finished" podID="49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" containerID="7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a" exitCode=0
Nov 03 13:44:02.902243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.892402    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m" event={"ID":"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80","Type":"ContainerDied","Data":"7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a"}
Nov 03 13:44:02.902243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.892443    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m" event={"ID":"49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80","Type":"ContainerDied","Data":"d24da4ca52c058adf325438134afbed6b056d12dc128d94c2fb9b8405212cc65"}
Nov 03 13:44:02.902243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.892471    5420 scope.go:122] "RemoveContainer" containerID="7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a"
Nov 03 13:44:02.902243 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:02.892639    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"
Nov 03 13:44:03.054827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.044275    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"]
Nov 03 13:44:03.061439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.061413    5420 scope.go:122] "RemoveContainer" containerID="7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a"
Nov 03 13:44:03.064897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.064871    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e440bc-85679c9659-sjt9m"]
Nov 03 13:44:03.066857 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:03.066833    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a\": container with ID starting with 7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a not found: ID does not exist" containerID="7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a"
Nov 03 13:44:03.067007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.066981    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a"} err="failed to get container status \"7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a\": rpc error: code = NotFound desc = could not find container \"7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a\": container with ID starting with 7ec66128eb008d36170a4fb558237a12a71d869dd94a37eb40142403f43a482a not found: ID does not exist"
Nov 03 13:44:03.072168 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.071843    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:44:03.091411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.091389    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" path="/var/lib/kubelet/pods/49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80/volumes"
Nov 03 13:44:03.265226 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.265191    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9eda720-9a95-420a-bc9b-22d9c080177d-kube-api-access-7ttlg\" (UniqueName: \"kubernetes.io/projected/d9eda720-9a95-420a-bc9b-22d9c080177d-kube-api-access-7ttlg\") pod \"d9eda720-9a95-420a-bc9b-22d9c080177d\" (UID: \"d9eda720-9a95-420a-bc9b-22d9c080177d\") "
Nov 03 13:44:03.265461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.265440    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9eda720-9a95-420a-bc9b-22d9c080177d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eda720-9a95-420a-bc9b-22d9c080177d-plugins-config-volume\") pod \"d9eda720-9a95-420a-bc9b-22d9c080177d\" (UID: \"d9eda720-9a95-420a-bc9b-22d9c080177d\") "
Nov 03 13:44:03.265834 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.265807    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9eda720-9a95-420a-bc9b-22d9c080177d-plugins-config-volume" pod "d9eda720-9a95-420a-bc9b-22d9c080177d" (UID: "d9eda720-9a95-420a-bc9b-22d9c080177d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:03.266140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.266116    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eda720-9a95-420a-bc9b-22d9c080177d-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:03.267587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.267564    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9eda720-9a95-420a-bc9b-22d9c080177d-kube-api-access-7ttlg" pod "d9eda720-9a95-420a-bc9b-22d9c080177d" (UID: "d9eda720-9a95-420a-bc9b-22d9c080177d"). InnerVolumeSpecName "kube-api-access-7ttlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:03.273849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.273832    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:44:03.288586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.288563    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:03.367010 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.366980    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-tmp-dir\") pod \"e1c82ef7-9eee-408d-8d33-0fd166280fba\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") "
Nov 03 13:44:03.367245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367023    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-dshm\") pod \"e1c82ef7-9eee-408d-8d33-0fd166280fba\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") "
Nov 03 13:44:03.367245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367138    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1c82ef7-9eee-408d-8d33-0fd166280fba-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c82ef7-9eee-408d-8d33-0fd166280fba-tls-certs\") pod \"e1c82ef7-9eee-408d-8d33-0fd166280fba\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") "
Nov 03 13:44:03.367245 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367197    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-model-cache\") pod \"e1c82ef7-9eee-408d-8d33-0fd166280fba\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") "
Nov 03 13:44:03.367431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367247    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c82ef7-9eee-408d-8d33-0fd166280fba-kube-api-access-m65vx\" (UniqueName: \"kubernetes.io/projected/e1c82ef7-9eee-408d-8d33-0fd166280fba-kube-api-access-m65vx\") pod \"e1c82ef7-9eee-408d-8d33-0fd166280fba\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") "
Nov 03 13:44:03.367431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367303    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-home\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-home\") pod \"e1c82ef7-9eee-408d-8d33-0fd166280fba\" (UID: \"e1c82ef7-9eee-408d-8d33-0fd166280fba\") "
Nov 03 13:44:03.367431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367322    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-tmp-dir" pod "e1c82ef7-9eee-408d-8d33-0fd166280fba" (UID: "e1c82ef7-9eee-408d-8d33-0fd166280fba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:03.367587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367457    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-model-cache" pod "e1c82ef7-9eee-408d-8d33-0fd166280fba" (UID: "e1c82ef7-9eee-408d-8d33-0fd166280fba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:03.367646 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367586    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-home" pod "e1c82ef7-9eee-408d-8d33-0fd166280fba" (UID: "e1c82ef7-9eee-408d-8d33-0fd166280fba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:03.367821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367801    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:03.367899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367827    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:03.367899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367841    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:03.367899 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.367856    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ttlg\" (UniqueName: \"kubernetes.io/projected/d9eda720-9a95-420a-bc9b-22d9c080177d-kube-api-access-7ttlg\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:03.369520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.369485    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-dshm" pod "e1c82ef7-9eee-408d-8d33-0fd166280fba" (UID: "e1c82ef7-9eee-408d-8d33-0fd166280fba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:03.369520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.369500    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c82ef7-9eee-408d-8d33-0fd166280fba-tls-certs" pod "e1c82ef7-9eee-408d-8d33-0fd166280fba" (UID: "e1c82ef7-9eee-408d-8d33-0fd166280fba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:03.369520 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.369506    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c82ef7-9eee-408d-8d33-0fd166280fba-kube-api-access-m65vx" pod "e1c82ef7-9eee-408d-8d33-0fd166280fba" (UID: "e1c82ef7-9eee-408d-8d33-0fd166280fba"). InnerVolumeSpecName "kube-api-access-m65vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:03.469028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.469001    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c82ef7-9eee-408d-8d33-0fd166280fba-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:03.469028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.469025    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c82ef7-9eee-408d-8d33-0fd166280fba-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:03.469241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.469035    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m65vx\" (UniqueName: \"kubernetes.io/projected/e1c82ef7-9eee-408d-8d33-0fd166280fba-kube-api-access-m65vx\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:03.530806 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.530779    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:03.900975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.900885    5420 generic.go:356] "Generic (PLEG): container finished" podID="e1c82ef7-9eee-408d-8d33-0fd166280fba" containerID="46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944" exitCode=0
Nov 03 13:44:03.900975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.900929    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"
Nov 03 13:44:03.901206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.900986    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx" event={"ID":"e1c82ef7-9eee-408d-8d33-0fd166280fba","Type":"ContainerDied","Data":"46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944"}
Nov 03 13:44:03.901206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.901031    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx" event={"ID":"e1c82ef7-9eee-408d-8d33-0fd166280fba","Type":"ContainerDied","Data":"8cae73f84ee64dad85e7cceb9b5d73d9fce190e63cebf6e543ad3289e86e9d59"}
Nov 03 13:44:03.901206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.901053    5420 scope.go:122] "RemoveContainer" containerID="46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944"
Nov 03 13:44:03.902624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.902603    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g" event={"ID":"d9eda720-9a95-420a-bc9b-22d9c080177d","Type":"ContainerDied","Data":"d59056afd5e9b78b998ec74c851e8db0b8ef17d55e528cc510766e53bd5f77ef"}
Nov 03 13:44:03.902752 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.902677    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"
Nov 03 13:44:03.911886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.911865    5420 scope.go:122] "RemoveContainer" containerID="46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944"
Nov 03 13:44:03.912317 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:03.912154    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944\": container with ID starting with 46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944 not found: ID does not exist" containerID="46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944"
Nov 03 13:44:03.912317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.912184    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944"} err="failed to get container status \"46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944\": rpc error: code = NotFound desc = could not find container \"46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944\": container with ID starting with 46b05db81c912b6130c33a7d6679c9f80902c807019e48e22f4f337bddbb9944 not found: ID does not exist"
Nov 03 13:44:03.912317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.912199    5420 scope.go:122] "RemoveContainer" containerID="fe1dd7eaf7f2deed185d8207164a6ab77a7d1db544bb472a6e001aff089e1d09"
Nov 03 13:44:03.929637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.929617    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"]
Nov 03 13:44:03.934274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.934255    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e440bc-7b86465fd7-q6p6g"]
Nov 03 13:44:03.951057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.951037    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"]
Nov 03 13:44:03.967391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.967370    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/rate-test-model-9add72-kserve-55486764f-2fgwx"]
Nov 03 13:44:03.967622 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:03.967606    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:04.152166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.152085    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:04.315319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.315206    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:04.537350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.537321    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:04.909449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.909370    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"]
Nov 03 13:44:04.909631 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.909559    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy" containerID="cri-o://73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e" gracePeriod=30
Nov 03 13:44:04.930072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.930046    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"]
Nov 03 13:44:04.930272 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.930249    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy" containerID="cri-o://1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec" gracePeriod=30
Nov 03 13:44:04.958853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.958817    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"]
Nov 03 13:44:04.959035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.959016    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy" containerID="cri-o://56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0" gracePeriod=30
Nov 03 13:44:04.962975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.962948    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"]
Nov 03 13:44:04.963161 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:04.963143    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy" containerID="cri-o://27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9" gracePeriod=30
Nov 03 13:44:05.054772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:05.054378    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:05.084338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:05.084306    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9eda720-9a95-420a-bc9b-22d9c080177d" path="/var/lib/kubelet/pods/d9eda720-9a95-420a-bc9b-22d9c080177d/volumes"
Nov 03 13:44:05.084794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:05.084780    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c82ef7-9eee-408d-8d33-0fd166280fba" path="/var/lib/kubelet/pods/e1c82ef7-9eee-408d-8d33-0fd166280fba/volumes"
Nov 03 13:44:05.433158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:05.433117    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:05.573004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:05.572970    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c" podUID="f289deff-b01c-46c1-9336-bc43d9a8023b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:44:06.228091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:06.228046    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:06.406416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:06.406385    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:06.659722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:06.659646    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:06.998604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:06.996885    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:07.254070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:07.253981    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:07.890443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:07.890391    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:08.050250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:08.050177    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:08.764509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:08.764476    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:09.137868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.137786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:09.302137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.302099    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:09.806707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.806673    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"]
Nov 03 13:44:09.807202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807189    5420 cpu_manager.go:414] "containerMap: removing container" podUID="49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" containerName="payload-processing"
Nov 03 13:44:09.807284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807203    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" containerName="payload-processing"
Nov 03 13:44:09.807284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807242    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e1c82ef7-9eee-408d-8d33-0fd166280fba" containerName="main"
Nov 03 13:44:09.807284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807248    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c82ef7-9eee-408d-8d33-0fd166280fba" containerName="main"
Nov 03 13:44:09.807284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807263    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d9eda720-9a95-420a-bc9b-22d9c080177d" containerName="payload-pre-processing"
Nov 03 13:44:09.807284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807268    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9eda720-9a95-420a-bc9b-22d9c080177d" containerName="payload-pre-processing"
Nov 03 13:44:09.807284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807277    5420 cpu_manager.go:414] "containerMap: removing container" podUID="00298a04-971e-4bc0-a25f-b0bd162681f9" containerName="revoke-keys"
Nov 03 13:44:09.807284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807282    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00298a04-971e-4bc0-a25f-b0bd162681f9" containerName="revoke-keys"
Nov 03 13:44:09.807526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807356    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c82ef7-9eee-408d-8d33-0fd166280fba" containerName="main"
Nov 03 13:44:09.807526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807367    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9eda720-9a95-420a-bc9b-22d9c080177d" containerName="payload-pre-processing"
Nov 03 13:44:09.807526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807375    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="00298a04-971e-4bc0-a25f-b0bd162681f9" containerName="revoke-keys"
Nov 03 13:44:09.807526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.807383    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d6d9b0-3097-4b04-b6f9-ebc9fa6c2a80" containerName="payload-processing"
Nov 03 13:44:09.810768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.810749    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:09.816997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.816982    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:44:09.826037 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.826019    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:44:09.834591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.834569    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:44:09.839632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.839609    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"]
Nov 03 13:44:09.841431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.841405    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjhq\" (UniqueName: \"kubernetes.io/projected/666592f1-c7c3-4d29-8ac3-2e8611d94826-kube-api-access-2wjhq\") pod \"maas-api-revoke-sub-2a09b7f6-4frs5\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:09.841580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.841562    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a09b7f6-4frs5\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:09.841668 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.841635    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a09b7f6-4frs5\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:09.846193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.846167    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:09.918160 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.918127    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:44:09.942720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.942698    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a09b7f6-4frs5\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:09.942803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.942744    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjhq\" (UniqueName: \"kubernetes.io/projected/666592f1-c7c3-4d29-8ac3-2e8611d94826-kube-api-access-2wjhq\") pod \"maas-api-revoke-sub-2a09b7f6-4frs5\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:09.942882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.942869    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a09b7f6-4frs5\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:09.943371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.943350    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a09b7f6-4frs5\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:09.945305 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.945290    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a09b7f6-4frs5\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:09.961554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:09.961530    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjhq\" (UniqueName: \"kubernetes.io/projected/666592f1-c7c3-4d29-8ac3-2e8611d94826-kube-api-access-2wjhq\") pod \"maas-api-revoke-sub-2a09b7f6-4frs5\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:10.125250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.124922    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:10.226868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.226536    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:44:10.269279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.259590    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-kube-api-access-tzgf8\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-kube-api-access-tzgf8\") pod \"1dd637f3-b50c-4f81-a105-c92294ad4e37\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") "
Nov 03 13:44:10.269279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.259663    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-token\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-token\") pod \"1dd637f3-b50c-4f81-a105-c92294ad4e37\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") "
Nov 03 13:44:10.269279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.259717    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-data\") pod \"1dd637f3-b50c-4f81-a105-c92294ad4e37\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") "
Nov 03 13:44:10.269279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.259820    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-socket\") pod \"1dd637f3-b50c-4f81-a105-c92294ad4e37\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") "
Nov 03 13:44:10.269279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.259869    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-certs\") pod \"1dd637f3-b50c-4f81-a105-c92294ad4e37\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") "
Nov 03 13:44:10.269279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.259895    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dd637f3-b50c-4f81-a105-c92294ad4e37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd637f3-b50c-4f81-a105-c92294ad4e37-istiod-ca-cert\") pod \"1dd637f3-b50c-4f81-a105-c92294ad4e37\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") "
Nov 03 13:44:10.269279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.259944    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-podinfo\") pod \"1dd637f3-b50c-4f81-a105-c92294ad4e37\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") "
Nov 03 13:44:10.269279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.259971    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-envoy\") pod \"1dd637f3-b50c-4f81-a105-c92294ad4e37\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") "
Nov 03 13:44:10.269279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.260043    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-credential-socket\") pod \"1dd637f3-b50c-4f81-a105-c92294ad4e37\" (UID: \"1dd637f3-b50c-4f81-a105-c92294ad4e37\") "
Nov 03 13:44:10.276627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.276278    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-data" pod "1dd637f3-b50c-4f81-a105-c92294ad4e37" (UID: "1dd637f3-b50c-4f81-a105-c92294ad4e37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.276627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.276375    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-certs" pod "1dd637f3-b50c-4f81-a105-c92294ad4e37" (UID: "1dd637f3-b50c-4f81-a105-c92294ad4e37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.276627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.276430    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-socket" pod "1dd637f3-b50c-4f81-a105-c92294ad4e37" (UID: "1dd637f3-b50c-4f81-a105-c92294ad4e37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.276627 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.276593    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd637f3-b50c-4f81-a105-c92294ad4e37-istiod-ca-cert" pod "1dd637f3-b50c-4f81-a105-c92294ad4e37" (UID: "1dd637f3-b50c-4f81-a105-c92294ad4e37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:10.277100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.277048    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-credential-socket" pod "1dd637f3-b50c-4f81-a105-c92294ad4e37" (UID: "1dd637f3-b50c-4f81-a105-c92294ad4e37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.290686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.290633    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-token" pod "1dd637f3-b50c-4f81-a105-c92294ad4e37" (UID: "1dd637f3-b50c-4f81-a105-c92294ad4e37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.293956 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.293911    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-podinfo" pod "1dd637f3-b50c-4f81-a105-c92294ad4e37" (UID: "1dd637f3-b50c-4f81-a105-c92294ad4e37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:10.295260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.294289    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-kube-api-access-tzgf8" pod "1dd637f3-b50c-4f81-a105-c92294ad4e37" (UID: "1dd637f3-b50c-4f81-a105-c92294ad4e37"). InnerVolumeSpecName "kube-api-access-tzgf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.299066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.299012    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-envoy" pod "1dd637f3-b50c-4f81-a105-c92294ad4e37" (UID: "1dd637f3-b50c-4f81-a105-c92294ad4e37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.331855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.331552    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:44:10.338431 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.338405    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:44:10.343363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.343331    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:44:10.361546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361520    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istiod-ca-cert\") pod \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") "
Nov 03 13:44:10.361693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361581    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-token\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-token\") pod \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") "
Nov 03 13:44:10.361693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361612    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-credential-socket\") pod \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") "
Nov 03 13:44:10.361693 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361667    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-data\") pod \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") "
Nov 03 13:44:10.361879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361730    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-kube-api-access-shqft\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-kube-api-access-shqft\") pod \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") "
Nov 03 13:44:10.361879 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361830    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-envoy\") pod \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") "
Nov 03 13:44:10.361996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361889    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-socket\") pod \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") "
Nov 03 13:44:10.361996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361921    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-certs\") pod \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") "
Nov 03 13:44:10.361996 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361986    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-podinfo\") pod \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\" (UID: \"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f\") "
Nov 03 13:44:10.362157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.361996    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-data" pod "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" (UID: "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.362157 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362012    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-credential-socket" pod "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" (UID: "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.362296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362181    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-socket" pod "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" (UID: "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.362296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362202    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-certs" pod "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" (UID: "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.362427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362405    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istiod-ca-cert" pod "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" (UID: "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:10.362492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362462    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362479    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362492    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362503    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362513    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362522    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzgf8\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-kube-api-access-tzgf8\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362535    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362548    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362558    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362567    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362575    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362583    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd637f3-b50c-4f81-a105-c92294ad4e37-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.362604 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.362596    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd637f3-b50c-4f81-a105-c92294ad4e37-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.364483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.364452    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-kube-api-access-shqft" pod "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" (UID: "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f"). InnerVolumeSpecName "kube-api-access-shqft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.364599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.364519    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-token" pod "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" (UID: "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.364599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.364545    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-envoy" pod "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" (UID: "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.365335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.365313    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-podinfo" pod "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" (UID: "5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:10.463772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.463745    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-data\") pod \"eb216de6-36b1-4280-b763-fb951b3bb9d6\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") "
Nov 03 13:44:10.463921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.463788    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-data\") pod \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") "
Nov 03 13:44:10.463921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.463822    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-envoy\") pod \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") "
Nov 03 13:44:10.463921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.463880    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb216de6-36b1-4280-b763-fb951b3bb9d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb216de6-36b1-4280-b763-fb951b3bb9d6-istiod-ca-cert\") pod \"eb216de6-36b1-4280-b763-fb951b3bb9d6\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") "
Nov 03 13:44:10.463921 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.463916    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istiod-ca-cert\") pod \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") "
Nov 03 13:44:10.464156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.463951    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-podinfo\") pod \"eb216de6-36b1-4280-b763-fb951b3bb9d6\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") "
Nov 03 13:44:10.464156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.463981    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-credential-socket\") pod \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") "
Nov 03 13:44:10.464156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464031    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-token\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-token\") pod \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") "
Nov 03 13:44:10.464156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464031    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-data" pod "eb216de6-36b1-4280-b763-fb951b3bb9d6" (UID: "eb216de6-36b1-4280-b763-fb951b3bb9d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.464156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.463914    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"]
Nov 03 13:44:10.464156 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464120    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-certs\") pod \"eb216de6-36b1-4280-b763-fb951b3bb9d6\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") "
Nov 03 13:44:10.464554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464165    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-socket\") pod \"eb216de6-36b1-4280-b763-fb951b3bb9d6\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") "
Nov 03 13:44:10.464554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464232    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-certs\") pod \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") "
Nov 03 13:44:10.464554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464282    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-socket\") pod \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") "
Nov 03 13:44:10.464554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464317    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-kube-api-access-pqhgd\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-kube-api-access-pqhgd\") pod \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") "
Nov 03 13:44:10.464554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464395    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-token\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-token\") pod \"eb216de6-36b1-4280-b763-fb951b3bb9d6\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") "
Nov 03 13:44:10.464554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464426    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-kube-api-access-d4s7r\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-kube-api-access-d4s7r\") pod \"eb216de6-36b1-4280-b763-fb951b3bb9d6\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") "
Nov 03 13:44:10.464554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464464    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb216de6-36b1-4280-b763-fb951b3bb9d6-istiod-ca-cert" pod "eb216de6-36b1-4280-b763-fb951b3bb9d6" (UID: "eb216de6-36b1-4280-b763-fb951b3bb9d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:10.464554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464473    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-podinfo\") pod \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\" (UID: \"d67dc242-8956-4153-9d6e-7cedaf3f5b73\") "
Nov 03 13:44:10.465099 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:44:10.464680    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666592f1_c7c3_4d29_8ac3_2e8611d94826.slice/crio-eb9b556d86c7a7dd2f5cccabfb8325e6bfecf2b787013d0f34cb69ea7e85c5ab WatchSource:0}: Error finding container eb9b556d86c7a7dd2f5cccabfb8325e6bfecf2b787013d0f34cb69ea7e85c5ab: Status 404 returned error can't find the container with id eb9b556d86c7a7dd2f5cccabfb8325e6bfecf2b787013d0f34cb69ea7e85c5ab
Nov 03 13:44:10.465099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464920    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istiod-ca-cert" pod "d67dc242-8956-4153-9d6e-7cedaf3f5b73" (UID: "d67dc242-8956-4153-9d6e-7cedaf3f5b73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:10.465099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.464966    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-envoy\") pod \"eb216de6-36b1-4280-b763-fb951b3bb9d6\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") "
Nov 03 13:44:10.465099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465004    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-certs" pod "d67dc242-8956-4153-9d6e-7cedaf3f5b73" (UID: "d67dc242-8956-4153-9d6e-7cedaf3f5b73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.465099 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465032    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-credential-socket\") pod \"eb216de6-36b1-4280-b763-fb951b3bb9d6\" (UID: \"eb216de6-36b1-4280-b763-fb951b3bb9d6\") "
Nov 03 13:44:10.465404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465229    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-credential-socket" pod "d67dc242-8956-4153-9d6e-7cedaf3f5b73" (UID: "d67dc242-8956-4153-9d6e-7cedaf3f5b73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.465404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465199    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-data" pod "d67dc242-8956-4153-9d6e-7cedaf3f5b73" (UID: "d67dc242-8956-4153-9d6e-7cedaf3f5b73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.465533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465443    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-credential-socket" pod "eb216de6-36b1-4280-b763-fb951b3bb9d6" (UID: "eb216de6-36b1-4280-b763-fb951b3bb9d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465677    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465706    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465720    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb216de6-36b1-4280-b763-fb951b3bb9d6-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465735    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465750    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465766    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqft\" (UniqueName: \"kubernetes.io/projected/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-kube-api-access-shqft\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465781    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465794    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.465807    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.466069    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.466092    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.466090    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-socket" pod "d67dc242-8956-4153-9d6e-7cedaf3f5b73" (UID: "d67dc242-8956-4153-9d6e-7cedaf3f5b73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.466107    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.466819    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-certs" pod "eb216de6-36b1-4280-b763-fb951b3bb9d6" (UID: "eb216de6-36b1-4280-b763-fb951b3bb9d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.467025    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-podinfo" pod "eb216de6-36b1-4280-b763-fb951b3bb9d6" (UID: "eb216de6-36b1-4280-b763-fb951b3bb9d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.467632    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-token" pod "d67dc242-8956-4153-9d6e-7cedaf3f5b73" (UID: "d67dc242-8956-4153-9d6e-7cedaf3f5b73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.467866    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-socket" pod "eb216de6-36b1-4280-b763-fb951b3bb9d6" (UID: "eb216de6-36b1-4280-b763-fb951b3bb9d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.468313    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-envoy" pod "eb216de6-36b1-4280-b763-fb951b3bb9d6" (UID: "eb216de6-36b1-4280-b763-fb951b3bb9d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.468403    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-kube-api-access-pqhgd" pod "d67dc242-8956-4153-9d6e-7cedaf3f5b73" (UID: "d67dc242-8956-4153-9d6e-7cedaf3f5b73"). InnerVolumeSpecName "kube-api-access-pqhgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.468905    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-token" pod "eb216de6-36b1-4280-b763-fb951b3bb9d6" (UID: "eb216de6-36b1-4280-b763-fb951b3bb9d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.468273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.469165    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-kube-api-access-d4s7r" pod "eb216de6-36b1-4280-b763-fb951b3bb9d6" (UID: "eb216de6-36b1-4280-b763-fb951b3bb9d6"). InnerVolumeSpecName "kube-api-access-d4s7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:10.470897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.469704    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-envoy" pod "d67dc242-8956-4153-9d6e-7cedaf3f5b73" (UID: "d67dc242-8956-4153-9d6e-7cedaf3f5b73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:10.475799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.475779    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-podinfo" pod "d67dc242-8956-4153-9d6e-7cedaf3f5b73" (UID: "d67dc242-8956-4153-9d6e-7cedaf3f5b73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:10.567197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567109    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567138    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4s7r\" (UniqueName: \"kubernetes.io/projected/eb216de6-36b1-4280-b763-fb951b3bb9d6-kube-api-access-d4s7r\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567153    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567166    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567177    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567190    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb216de6-36b1-4280-b763-fb951b3bb9d6-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567202    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567228    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567243    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb216de6-36b1-4280-b763-fb951b3bb9d6-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567255    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67dc242-8956-4153-9d6e-7cedaf3f5b73-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.567546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.567267    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqhgd\" (UniqueName: \"kubernetes.io/projected/d67dc242-8956-4153-9d6e-7cedaf3f5b73-kube-api-access-pqhgd\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:10.645826 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.645795    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:10.971559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.971527    5420 generic.go:356] "Generic (PLEG): container finished" podID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerID="1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec" exitCode=0
Nov 03 13:44:10.971709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.971582    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"
Nov 03 13:44:10.971709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.971608    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" event={"ID":"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f","Type":"ContainerDied","Data":"1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec"}
Nov 03 13:44:10.971709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.971661    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n" event={"ID":"5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f","Type":"ContainerDied","Data":"38c691e7476fb637649e97db52146b53f4d705819cb9bd30bb70eeebd3e4424c"}
Nov 03 13:44:10.971709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.971684    5420 scope.go:122] "RemoveContainer" containerID="1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec"
Nov 03 13:44:10.973147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.973124    5420 generic.go:356] "Generic (PLEG): container finished" podID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerID="27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9" exitCode=0
Nov 03 13:44:10.973297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.973172    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"
Nov 03 13:44:10.973371 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.973206    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" event={"ID":"eb216de6-36b1-4280-b763-fb951b3bb9d6","Type":"ContainerDied","Data":"27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9"}
Nov 03 13:44:10.973424 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.973371    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r" event={"ID":"eb216de6-36b1-4280-b763-fb951b3bb9d6","Type":"ContainerDied","Data":"a752f1f69b5bec658596900f506b9d0e9cf3109806b392a7e31ab8ca1c5f37f3"}
Nov 03 13:44:10.975249 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.975223    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5" event={"ID":"666592f1-c7c3-4d29-8ac3-2e8611d94826","Type":"ContainerStarted","Data":"b85280e67600a7d01b61cdb738b8e7842d287e553d5e451467bc58361f1590ac"}
Nov 03 13:44:10.975374 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.975269    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5" event={"ID":"666592f1-c7c3-4d29-8ac3-2e8611d94826","Type":"ContainerStarted","Data":"eb9b556d86c7a7dd2f5cccabfb8325e6bfecf2b787013d0f34cb69ea7e85c5ab"}
Nov 03 13:44:10.976959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.976937    5420 generic.go:356] "Generic (PLEG): container finished" podID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerID="73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e" exitCode=0
Nov 03 13:44:10.977074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.976979    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"
Nov 03 13:44:10.977176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.977065    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" event={"ID":"1dd637f3-b50c-4f81-a105-c92294ad4e37","Type":"ContainerDied","Data":"73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e"}
Nov 03 13:44:10.977176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.977120    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j" event={"ID":"1dd637f3-b50c-4f81-a105-c92294ad4e37","Type":"ContainerDied","Data":"7111c8042fa5d445eca1c611c4f2dd78bcd869436e348c7d5f7df4cd67c521ba"}
Nov 03 13:44:10.978878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.978856    5420 generic.go:356] "Generic (PLEG): container finished" podID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerID="56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0" exitCode=0
Nov 03 13:44:10.978982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.978894    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"
Nov 03 13:44:10.978982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.978975    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" event={"ID":"d67dc242-8956-4153-9d6e-7cedaf3f5b73","Type":"ContainerDied","Data":"56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0"}
Nov 03 13:44:10.979134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.979010    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7" event={"ID":"d67dc242-8956-4153-9d6e-7cedaf3f5b73","Type":"ContainerDied","Data":"6bc9fdcfcaea0b866481f411c94c7d3f8814664f8236e1c30d3cd30d49c018e3"}
Nov 03 13:44:10.985309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.985226    5420 scope.go:122] "RemoveContainer" containerID="1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec"
Nov 03 13:44:10.985518 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:10.985497    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec\": container with ID starting with 1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec not found: ID does not exist" containerID="1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec"
Nov 03 13:44:10.985614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.985531    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec"} err="failed to get container status \"1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec\": rpc error: code = NotFound desc = could not find container \"1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec\": container with ID starting with 1e62cbe07e4c79c4bee063ca066f9dccece51d4392ac45cb608048f0b162e2ec not found: ID does not exist"
Nov 03 13:44:10.985614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.985552    5420 scope.go:122] "RemoveContainer" containerID="27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9"
Nov 03 13:44:10.997069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.997052    5420 scope.go:122] "RemoveContainer" containerID="27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9"
Nov 03 13:44:10.997346 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:10.997327    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9\": container with ID starting with 27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9 not found: ID does not exist" containerID="27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9"
Nov 03 13:44:10.997409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.997353    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9"} err="failed to get container status \"27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9\": rpc error: code = NotFound desc = could not find container \"27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9\": container with ID starting with 27beea3ad51dfd102fdf570b4f593ce76fcab8d3855453b644a88d3d0eb615d9 not found: ID does not exist"
Nov 03 13:44:10.997409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:10.997371    5420 scope.go:122] "RemoveContainer" containerID="73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e"
Nov 03 13:44:11.006290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.006274    5420 scope.go:122] "RemoveContainer" containerID="73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e"
Nov 03 13:44:11.006516 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:11.006500    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e\": container with ID starting with 73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e not found: ID does not exist" containerID="73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e"
Nov 03 13:44:11.006577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.006522    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e"} err="failed to get container status \"73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e\": rpc error: code = NotFound desc = could not find container \"73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e\": container with ID starting with 73fe4a824f87631cde6b590fa5950152046b099f4c471c7cc49159ef288a3a3e not found: ID does not exist"
Nov 03 13:44:11.006577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.006535    5420 scope.go:122] "RemoveContainer" containerID="56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0"
Nov 03 13:44:11.015538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.015494    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5" podStartSLOduration=2.015481996 podStartE2EDuration="2.015481996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:44:11.011891607 +0000 UTC m=+2092.089656653" watchObservedRunningTime="2026-11-03 13:44:11.015481996 +0000 UTC m=+2092.093247042"
Nov 03 13:44:11.015848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.015824    5420 scope.go:122] "RemoveContainer" containerID="56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0"
Nov 03 13:44:11.016110 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:11.016091    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0\": container with ID starting with 56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0 not found: ID does not exist" containerID="56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0"
Nov 03 13:44:11.016159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.016116    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0"} err="failed to get container status \"56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0\": rpc error: code = NotFound desc = could not find container \"56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0\": container with ID starting with 56bc39c02c6cde923af7c204b5783d79113e7ba807aeeb598a37dfa3e7a49cb0 not found: ID does not exist"
Nov 03 13:44:11.041679 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.041656    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"]
Nov 03 13:44:11.044794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.044769    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-ckn9r"]
Nov 03 13:44:11.061233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.061198    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"]
Nov 03 13:44:11.066964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.066944    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-gjjk7"]
Nov 03 13:44:11.084648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.084627    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" path="/var/lib/kubelet/pods/d67dc242-8956-4153-9d6e-7cedaf3f5b73/volumes"
Nov 03 13:44:11.085066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.085052    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" path="/var/lib/kubelet/pods/eb216de6-36b1-4280-b763-fb951b3bb9d6/volumes"
Nov 03 13:44:11.085364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.085352    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"]
Nov 03 13:44:11.090373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.090340    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-xsm6n"]
Nov 03 13:44:11.105075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.105050    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"]
Nov 03 13:44:11.111268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.111244    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-t4h6j"]
Nov 03 13:44:11.461030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.460995    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:11.991073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.991036    5420 generic.go:356] "Generic (PLEG): container finished" podID="666592f1-c7c3-4d29-8ac3-2e8611d94826" containerID="b85280e67600a7d01b61cdb738b8e7842d287e553d5e451467bc58361f1590ac" exitCode=0
Nov 03 13:44:11.991260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:11.991121    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5" event={"ID":"666592f1-c7c3-4d29-8ac3-2e8611d94826","Type":"ContainerDied","Data":"b85280e67600a7d01b61cdb738b8e7842d287e553d5e451467bc58361f1590ac"}
Nov 03 13:44:12.207849 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:12.207822    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:12.658730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:12.658702    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:12.905891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:12.905863    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:13.086320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.086285    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:13.092919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.092885    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" path="/var/lib/kubelet/pods/1dd637f3-b50c-4f81-a105-c92294ad4e37/volumes"
Nov 03 13:44:13.094030 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.093999    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" path="/var/lib/kubelet/pods/5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f/volumes"
Nov 03 13:44:13.158042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.158021    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:13.196408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.196382    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-xks-ca\") pod \"666592f1-c7c3-4d29-8ac3-2e8611d94826\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") "
Nov 03 13:44:13.196551 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.196469    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-openshift-ca\") pod \"666592f1-c7c3-4d29-8ac3-2e8611d94826\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") "
Nov 03 13:44:13.196649 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.196634    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666592f1-c7c3-4d29-8ac3-2e8611d94826-kube-api-access-2wjhq\" (UniqueName: \"kubernetes.io/projected/666592f1-c7c3-4d29-8ac3-2e8611d94826-kube-api-access-2wjhq\") pod \"666592f1-c7c3-4d29-8ac3-2e8611d94826\" (UID: \"666592f1-c7c3-4d29-8ac3-2e8611d94826\") "
Nov 03 13:44:13.197264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.197238    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-openshift-ca" pod "666592f1-c7c3-4d29-8ac3-2e8611d94826" (UID: "666592f1-c7c3-4d29-8ac3-2e8611d94826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:13.198640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.198612    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-xks-ca" pod "666592f1-c7c3-4d29-8ac3-2e8611d94826" (UID: "666592f1-c7c3-4d29-8ac3-2e8611d94826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:13.198844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.198829    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666592f1-c7c3-4d29-8ac3-2e8611d94826-kube-api-access-2wjhq" pod "666592f1-c7c3-4d29-8ac3-2e8611d94826" (UID: "666592f1-c7c3-4d29-8ac3-2e8611d94826"). InnerVolumeSpecName "kube-api-access-2wjhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:13.297871 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.297809    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wjhq\" (UniqueName: \"kubernetes.io/projected/666592f1-c7c3-4d29-8ac3-2e8611d94826-kube-api-access-2wjhq\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:13.297871 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.297836    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:13.297871 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.297848    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666592f1-c7c3-4d29-8ac3-2e8611d94826-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:13.971599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:13.971568    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:14.008664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:14.008635    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"
Nov 03 13:44:14.008805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:14.008666    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5" event={"ID":"666592f1-c7c3-4d29-8ac3-2e8611d94826","Type":"ContainerDied","Data":"eb9b556d86c7a7dd2f5cccabfb8325e6bfecf2b787013d0f34cb69ea7e85c5ab"}
Nov 03 13:44:14.008805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:14.008713    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb9b556d86c7a7dd2f5cccabfb8325e6bfecf2b787013d0f34cb69ea7e85c5ab"
Nov 03 13:44:14.039324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:14.039300    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"]
Nov 03 13:44:14.048920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:14.048899    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f98368b-49lh9"]
Nov 03 13:44:14.846778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:14.846744    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"]
Nov 03 13:44:14.941603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:14.941568    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:15.084167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:15.084136    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7d3b619-2ccb-4e8c-9c56-d05553bd56df" path="/var/lib/kubelet/pods/c7d3b619-2ccb-4e8c-9c56-d05553bd56df/volumes"
Nov 03 13:44:15.097063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:15.096998    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:15.572896 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:15.572861    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c" podUID="f289deff-b01c-46c1-9336-bc43d9a8023b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:44:15.735967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:15.735933    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:16.601733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:16.597777    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:16.782886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:16.782854    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:17.063666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:17.063634    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:44:17.323253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:17.323119    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:18.011564 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:18.011525    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:18.559115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:18.559086    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:18.877690 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:18.877611    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:19.607379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:19.607348    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:44:19.620666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:19.620644    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:44:19.627548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:19.627516    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:19.797545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:19.797518    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:19.847893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:19.847866    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl" podUID="8285a1b2-2969-4bdc-8385-aede1a87dbb6" containerName="payload-pre-processing" containerID="cri-o://eb46fdd87d879647b87c3a3d1391d3d982dce124c52424e2a4522ad894ea5229" gracePeriod=25
Nov 03 13:44:20.052893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.052863    5420 generic.go:356] "Generic (PLEG): container finished" podID="8285a1b2-2969-4bdc-8385-aede1a87dbb6" containerID="eb46fdd87d879647b87c3a3d1391d3d982dce124c52424e2a4522ad894ea5229" exitCode=0
Nov 03 13:44:20.053085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.052945    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl" event={"ID":"8285a1b2-2969-4bdc-8385-aede1a87dbb6","Type":"ContainerDied","Data":"eb46fdd87d879647b87c3a3d1391d3d982dce124c52424e2a4522ad894ea5229"}
Nov 03 13:44:20.106722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.106700    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:44:20.171047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.170982    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8285a1b2-2969-4bdc-8385-aede1a87dbb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285a1b2-2969-4bdc-8385-aede1a87dbb6-plugins-config-volume\") pod \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\" (UID: \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\") "
Nov 03 13:44:20.171185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.171098    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8285a1b2-2969-4bdc-8385-aede1a87dbb6-kube-api-access-j5nqh\" (UniqueName: \"kubernetes.io/projected/8285a1b2-2969-4bdc-8385-aede1a87dbb6-kube-api-access-j5nqh\") pod \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\" (UID: \"8285a1b2-2969-4bdc-8385-aede1a87dbb6\") "
Nov 03 13:44:20.171414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.171392    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8285a1b2-2969-4bdc-8385-aede1a87dbb6-plugins-config-volume" pod "8285a1b2-2969-4bdc-8385-aede1a87dbb6" (UID: "8285a1b2-2969-4bdc-8385-aede1a87dbb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:20.171634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.171615    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285a1b2-2969-4bdc-8385-aede1a87dbb6-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:20.173085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.173061    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8285a1b2-2969-4bdc-8385-aede1a87dbb6-kube-api-access-j5nqh" pod "8285a1b2-2969-4bdc-8385-aede1a87dbb6" (UID: "8285a1b2-2969-4bdc-8385-aede1a87dbb6"). InnerVolumeSpecName "kube-api-access-j5nqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:20.272993 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.272963    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5nqh\" (UniqueName: \"kubernetes.io/projected/8285a1b2-2969-4bdc-8385-aede1a87dbb6-kube-api-access-j5nqh\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:20.383878 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.383839    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:20.939418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:20.939384    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"]
Nov 03 13:44:21.006382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.006350    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:21.062337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.062309    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl" event={"ID":"8285a1b2-2969-4bdc-8385-aede1a87dbb6","Type":"ContainerDied","Data":"d566f711c76e0adbba2da0a8f0c6de050edb2640e680cf451d4adb27934af3d4"}
Nov 03 13:44:21.062337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.062332    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"
Nov 03 13:44:21.062553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.062368    5420 scope.go:122] "RemoveContainer" containerID="eb46fdd87d879647b87c3a3d1391d3d982dce124c52424e2a4522ad894ea5229"
Nov 03 13:44:21.095451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.095427    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"]
Nov 03 13:44:21.102916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.102882    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9f71f2-6d44d7746c-vfxjl"]
Nov 03 13:44:21.173094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.173051    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:21.476963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.476901    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:44:21.598138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.598110    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f289deff-b01c-46c1-9336-bc43d9a8023b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f289deff-b01c-46c1-9336-bc43d9a8023b-plugins-config-volume\") pod \"f289deff-b01c-46c1-9336-bc43d9a8023b\" (UID: \"f289deff-b01c-46c1-9336-bc43d9a8023b\") "
Nov 03 13:44:21.598345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.598202    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f289deff-b01c-46c1-9336-bc43d9a8023b-kube-api-access-p8zq8\" (UniqueName: \"kubernetes.io/projected/f289deff-b01c-46c1-9336-bc43d9a8023b-kube-api-access-p8zq8\") pod \"f289deff-b01c-46c1-9336-bc43d9a8023b\" (UID: \"f289deff-b01c-46c1-9336-bc43d9a8023b\") "
Nov 03 13:44:21.598553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.598531    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f289deff-b01c-46c1-9336-bc43d9a8023b-plugins-config-volume" pod "f289deff-b01c-46c1-9336-bc43d9a8023b" (UID: "f289deff-b01c-46c1-9336-bc43d9a8023b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:21.600187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.600166    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f289deff-b01c-46c1-9336-bc43d9a8023b-kube-api-access-p8zq8" pod "f289deff-b01c-46c1-9336-bc43d9a8023b" (UID: "f289deff-b01c-46c1-9336-bc43d9a8023b"). InnerVolumeSpecName "kube-api-access-p8zq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:21.678247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.678221    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:21.699584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.699559    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f289deff-b01c-46c1-9336-bc43d9a8023b-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:21.699584 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.699582    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8zq8\" (UniqueName: \"kubernetes.io/projected/f289deff-b01c-46c1-9336-bc43d9a8023b-kube-api-access-p8zq8\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:21.834936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:21.834859    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:22.070026 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.069992    5420 generic.go:356] "Generic (PLEG): container finished" podID="f289deff-b01c-46c1-9336-bc43d9a8023b" containerID="1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2" exitCode=0
Nov 03 13:44:22.070502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.070035    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"
Nov 03 13:44:22.070502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.070062    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c" event={"ID":"f289deff-b01c-46c1-9336-bc43d9a8023b","Type":"ContainerDied","Data":"1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2"}
Nov 03 13:44:22.070502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.070133    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c" event={"ID":"f289deff-b01c-46c1-9336-bc43d9a8023b","Type":"ContainerDied","Data":"c66f1440d59bb576e4ed971f82b1e268d4dcae326ee1b7ee84fbfc93e06ad867"}
Nov 03 13:44:22.070502 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.070155    5420 scope.go:122] "RemoveContainer" containerID="1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2"
Nov 03 13:44:22.081118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.081104    5420 scope.go:122] "RemoveContainer" containerID="1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2"
Nov 03 13:44:22.081375 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:22.081358    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2\": container with ID starting with 1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2 not found: ID does not exist" containerID="1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2"
Nov 03 13:44:22.081429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.081383    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2"} err="failed to get container status \"1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2\": rpc error: code = NotFound desc = could not find container \"1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2\": container with ID starting with 1d2d724e7d353a5c5e6e007c50343c24289757599a5c5532cf411828a0d8eaa2 not found: ID does not exist"
Nov 03 13:44:22.106159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.106108    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"]
Nov 03 13:44:22.114257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.114235    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c47670-576b8cbd4c-6p65c"]
Nov 03 13:44:22.537472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.537434    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:22.767599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.767562    5420 scope.go:122] "RemoveContainer" containerID="82ae1f762e8e01e5f52a85188b44515ce9162b3be487469d539400ccf2e71ca2"
Nov 03 13:44:22.782289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.782258    5420 scope.go:122] "RemoveContainer" containerID="1992d15455eb9535b75d99f76fb796213b8d426f1cd979fc652a6e367b41a311"
Nov 03 13:44:22.801179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:22.801043    5420 scope.go:122] "RemoveContainer" containerID="26701db19b4e5241fe6aae47ba48e77b400bd6c9e7d2d0ec19aa8e4d146ac0f4"
Nov 03 13:44:23.084123 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:23.084050    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8285a1b2-2969-4bdc-8385-aede1a87dbb6" path="/var/lib/kubelet/pods/8285a1b2-2969-4bdc-8385-aede1a87dbb6/volumes"
Nov 03 13:44:23.084516 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:23.084487    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f289deff-b01c-46c1-9336-bc43d9a8023b" path="/var/lib/kubelet/pods/f289deff-b01c-46c1-9336-bc43d9a8023b/volumes"
Nov 03 13:44:23.153995 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:23.153966    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:23.370746 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:23.370643    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:23.579221 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:23.579161    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:23.754410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:23.754375    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:23.940556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:23.940523    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"]
Nov 03 13:44:23.940770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:23.940748    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy" containerID="cri-o://59ed8135f6896468e3eb0485ce30397bd3de5967e4d8cb15f872ba71e3ecc4c3" gracePeriod=30
Nov 03 13:44:24.060739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:24.060656    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:24.545675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:24.545643    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:24.730270 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:24.730237    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:44:24.857873 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:24.857797    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:25.235108 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:25.235074    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:25.886832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:25.886735    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:25.940491 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:25.940447    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z" podUID="b40070c5-a92a-4a71-96d9-12fdc971c2fd" containerName="payload-processing" containerID="cri-o://78ce78ad0c76d490165064a27ef5e1cd89d14174df53301c4d112c6a2fa0ff80" gracePeriod=25
Nov 03 13:44:26.040456 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.040426    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"]
Nov 03 13:44:26.052302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.052275    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd21913-xkds4"]
Nov 03 13:44:26.100548 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.100520    5420 generic.go:356] "Generic (PLEG): container finished" podID="b40070c5-a92a-4a71-96d9-12fdc971c2fd" containerID="78ce78ad0c76d490165064a27ef5e1cd89d14174df53301c4d112c6a2fa0ff80" exitCode=0
Nov 03 13:44:26.100716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.100589    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z" event={"ID":"b40070c5-a92a-4a71-96d9-12fdc971c2fd","Type":"ContainerDied","Data":"78ce78ad0c76d490165064a27ef5e1cd89d14174df53301c4d112c6a2fa0ff80"}
Nov 03 13:44:26.175191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.175152    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:26.197838 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.197817    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:44:26.340794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.340738    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:26.353586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.353561    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-kube-api-access-brvwz\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-kube-api-access-brvwz\") pod \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") "
Nov 03 13:44:26.353726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.353632    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-provider-credentials\") pod \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") "
Nov 03 13:44:26.353787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.353775    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-plugins-config-volume\") pod \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") "
Nov 03 13:44:26.353823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.353797    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-routing-overlay-volume\") pod \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\" (UID: \"b40070c5-a92a-4a71-96d9-12fdc971c2fd\") "
Nov 03 13:44:26.354175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.354144    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-plugins-config-volume" pod "b40070c5-a92a-4a71-96d9-12fdc971c2fd" (UID: "b40070c5-a92a-4a71-96d9-12fdc971c2fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:26.354332 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.354278    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-routing-overlay-volume" pod "b40070c5-a92a-4a71-96d9-12fdc971c2fd" (UID: "b40070c5-a92a-4a71-96d9-12fdc971c2fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:26.355824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.355804    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-kube-api-access-brvwz" pod "b40070c5-a92a-4a71-96d9-12fdc971c2fd" (UID: "b40070c5-a92a-4a71-96d9-12fdc971c2fd"). InnerVolumeSpecName "kube-api-access-brvwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:26.355824 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.355815    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-provider-credentials" pod "b40070c5-a92a-4a71-96d9-12fdc971c2fd" (UID: "b40070c5-a92a-4a71-96d9-12fdc971c2fd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:26.455463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.455441    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:26.455463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.455465    5420 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b40070c5-a92a-4a71-96d9-12fdc971c2fd-routing-overlay-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:26.455611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.455475    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvwz\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-kube-api-access-brvwz\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:26.455611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.455486    5420 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b40070c5-a92a-4a71-96d9-12fdc971c2fd-provider-credentials\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:26.737717 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:26.737635    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:27.083432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:27.083361    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf4ddf1-b7c5-4fc1-8bee-be892405db47" path="/var/lib/kubelet/pods/7cf4ddf1-b7c5-4fc1-8bee-be892405db47/volumes"
Nov 03 13:44:27.108054 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:27.108022    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"
Nov 03 13:44:27.108171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:27.108057    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z" event={"ID":"b40070c5-a92a-4a71-96d9-12fdc971c2fd","Type":"ContainerDied","Data":"8e94382b08ee5cc863d8792f81b27f13ca298a9b454862a8ec8b183a9fc6908d"}
Nov 03 13:44:27.108171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:27.108115    5420 scope.go:122] "RemoveContainer" containerID="78ce78ad0c76d490165064a27ef5e1cd89d14174df53301c4d112c6a2fa0ff80"
Nov 03 13:44:27.152347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:27.152324    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"]
Nov 03 13:44:27.159890 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:27.159869    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-569cw928z"]
Nov 03 13:44:27.539523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:27.539493    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:28.233809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:28.233778    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:28.969885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:28.969854    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:44:29.063559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.063532    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:29.082987 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.082956    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40070c5-a92a-4a71-96d9-12fdc971c2fd" path="/var/lib/kubelet/pods/b40070c5-a92a-4a71-96d9-12fdc971c2fd/volumes"
Nov 03 13:44:29.126742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.126715    5420 generic.go:356] "Generic (PLEG): container finished" podID="52845948-0621-4d3a-895d-b57495762003" containerID="59ed8135f6896468e3eb0485ce30397bd3de5967e4d8cb15f872ba71e3ecc4c3" exitCode=0
Nov 03 13:44:29.126882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.126832    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" event={"ID":"52845948-0621-4d3a-895d-b57495762003","Type":"ContainerDied","Data":"59ed8135f6896468e3eb0485ce30397bd3de5967e4d8cb15f872ba71e3ecc4c3"}
Nov 03 13:44:29.126960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.126907    5420 scope.go:122] "RemoveContainer" containerID="2888860214ac067a034256b03f867aed3bf06663f415604f7c3c73638e29e0e6"
Nov 03 13:44:29.195769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.195744    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:44:29.284989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.284960    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52845948-0621-4d3a-895d-b57495762003-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52845948-0621-4d3a-895d-b57495762003-istio-podinfo\") pod \"52845948-0621-4d3a-895d-b57495762003\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") "
Nov 03 13:44:29.285496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285013    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-certs\") pod \"52845948-0621-4d3a-895d-b57495762003\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") "
Nov 03 13:44:29.285496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285063    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-data\") pod \"52845948-0621-4d3a-895d-b57495762003\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") "
Nov 03 13:44:29.285496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285122    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-envoy\") pod \"52845948-0621-4d3a-895d-b57495762003\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") "
Nov 03 13:44:29.285496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285163    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-credential-socket\") pod \"52845948-0621-4d3a-895d-b57495762003\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") "
Nov 03 13:44:29.285496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285290    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-istio-token\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-istio-token\") pod \"52845948-0621-4d3a-895d-b57495762003\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") "
Nov 03 13:44:29.285496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285344    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-socket\") pod \"52845948-0621-4d3a-895d-b57495762003\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") "
Nov 03 13:44:29.285496 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285388    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52845948-0621-4d3a-895d-b57495762003-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52845948-0621-4d3a-895d-b57495762003-istiod-ca-cert\") pod \"52845948-0621-4d3a-895d-b57495762003\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") "
Nov 03 13:44:29.285930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285413    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-data" pod "52845948-0621-4d3a-895d-b57495762003" (UID: "52845948-0621-4d3a-895d-b57495762003"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:29.285930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285422    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-certs" pod "52845948-0621-4d3a-895d-b57495762003" (UID: "52845948-0621-4d3a-895d-b57495762003"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:29.285930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285492    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-credential-socket" pod "52845948-0621-4d3a-895d-b57495762003" (UID: "52845948-0621-4d3a-895d-b57495762003"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:29.285930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285675    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-socket" pod "52845948-0621-4d3a-895d-b57495762003" (UID: "52845948-0621-4d3a-895d-b57495762003"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:29.285930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285446    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-kube-api-access-vrx48\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-kube-api-access-vrx48\") pod \"52845948-0621-4d3a-895d-b57495762003\" (UID: \"52845948-0621-4d3a-895d-b57495762003\") "
Nov 03 13:44:29.285930 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.285759    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52845948-0621-4d3a-895d-b57495762003-istiod-ca-cert" pod "52845948-0621-4d3a-895d-b57495762003" (UID: "52845948-0621-4d3a-895d-b57495762003"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:29.286315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.286143    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:29.286315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.286158    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:29.286315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.286168    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52845948-0621-4d3a-895d-b57495762003-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:29.286315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.286177    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:29.286315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.286186    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:29.287445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.287419    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-envoy" pod "52845948-0621-4d3a-895d-b57495762003" (UID: "52845948-0621-4d3a-895d-b57495762003"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:29.287801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.287770    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-kube-api-access-vrx48" pod "52845948-0621-4d3a-895d-b57495762003" (UID: "52845948-0621-4d3a-895d-b57495762003"). InnerVolumeSpecName "kube-api-access-vrx48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:29.287891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.287826    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52845948-0621-4d3a-895d-b57495762003-istio-podinfo" pod "52845948-0621-4d3a-895d-b57495762003" (UID: "52845948-0621-4d3a-895d-b57495762003"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:29.287891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.287846    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-istio-token" pod "52845948-0621-4d3a-895d-b57495762003" (UID: "52845948-0621-4d3a-895d-b57495762003"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:29.387267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.387236    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52845948-0621-4d3a-895d-b57495762003-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:29.387267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.387262    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:29.387267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.387277    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrx48\" (UniqueName: \"kubernetes.io/projected/52845948-0621-4d3a-895d-b57495762003-kube-api-access-vrx48\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:29.387590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:29.387288    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52845948-0621-4d3a-895d-b57495762003-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:30.006277 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:30.006244    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:30.134863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:30.134835    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"
Nov 03 13:44:30.135071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:30.134887    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x" event={"ID":"52845948-0621-4d3a-895d-b57495762003","Type":"ContainerDied","Data":"1800b7a4f282949b10a87dd02577ebbaaf8f504f0ed5ec47a61116b86abd8d01"}
Nov 03 13:44:30.135071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:30.134951    5420 scope.go:122] "RemoveContainer" containerID="59ed8135f6896468e3eb0485ce30397bd3de5967e4d8cb15f872ba71e3ecc4c3"
Nov 03 13:44:30.166244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:30.166221    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"]
Nov 03 13:44:30.172595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:30.172574    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-9f71f2-openshift-default-7bdc6f944f-cqq6x"]
Nov 03 13:44:30.604991 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:30.604959    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:30.842895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:30.842868    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:31.082999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.082962    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52845948-0621-4d3a-895d-b57495762003" path="/var/lib/kubelet/pods/52845948-0621-4d3a-895d-b57495762003/volumes"
Nov 03 13:44:31.111116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111082    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"]
Nov 03 13:44:31.111722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111704    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b40070c5-a92a-4a71-96d9-12fdc971c2fd" containerName="payload-processing"
Nov 03 13:44:31.111779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111724    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40070c5-a92a-4a71-96d9-12fdc971c2fd" containerName="payload-processing"
Nov 03 13:44:31.111779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111735    5420 cpu_manager.go:414] "containerMap: removing container" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy"
Nov 03 13:44:31.111779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111743    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy"
Nov 03 13:44:31.111779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111754    5420 cpu_manager.go:414] "containerMap: removing container" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy"
Nov 03 13:44:31.111779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111761    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy"
Nov 03 13:44:31.111779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111778    5420 cpu_manager.go:414] "containerMap: removing container" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111785    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111797    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f289deff-b01c-46c1-9336-bc43d9a8023b" containerName="payload-pre-processing"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111804    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f289deff-b01c-46c1-9336-bc43d9a8023b" containerName="payload-pre-processing"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111816    5420 cpu_manager.go:414] "containerMap: removing container" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111822    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111830    5420 cpu_manager.go:414] "containerMap: removing container" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111837    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111878    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8285a1b2-2969-4bdc-8385-aede1a87dbb6" containerName="payload-pre-processing"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111884    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8285a1b2-2969-4bdc-8385-aede1a87dbb6" containerName="payload-pre-processing"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111907    5420 cpu_manager.go:414] "containerMap: removing container" podUID="666592f1-c7c3-4d29-8ac3-2e8611d94826" containerName="revoke-keys"
Nov 03 13:44:31.111980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.111913    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666592f1-c7c3-4d29-8ac3-2e8611d94826" containerName="revoke-keys"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112001    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8285a1b2-2969-4bdc-8385-aede1a87dbb6" containerName="payload-pre-processing"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112011    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112021    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="666592f1-c7c3-4d29-8ac3-2e8611d94826" containerName="revoke-keys"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112036    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40070c5-a92a-4a71-96d9-12fdc971c2fd" containerName="payload-processing"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112044    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd637f3-b50c-4f81-a105-c92294ad4e37" containerName="istio-proxy"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112052    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67dc242-8956-4153-9d6e-7cedaf3f5b73" containerName="istio-proxy"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112062    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb216de6-36b1-4280-b763-fb951b3bb9d6" containerName="istio-proxy"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112070    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f289deff-b01c-46c1-9336-bc43d9a8023b" containerName="payload-pre-processing"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112077    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bab1e71-50d2-48a2-bdf7-c5f7a5bee42f" containerName="istio-proxy"
Nov 03 13:44:31.112349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.112084    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy"
Nov 03 13:44:31.116974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.116956    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.125553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.125461    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:44:31.125825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.125789    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:44:31.125913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.125834    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:44:31.125913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.125901    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:44:31.138352 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.138330    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"]
Nov 03 13:44:31.206452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.206423    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a24258-98rwv\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.206615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.206577    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd25t\" (UniqueName: \"kubernetes.io/projected/8eb73c91-ae95-4d53-91bf-27ae02c854ce-kube-api-access-fd25t\") pod \"maas-api-revoke-sub-a1a24258-98rwv\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.206686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.206635    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a24258-98rwv\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.308028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.307996    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd25t\" (UniqueName: \"kubernetes.io/projected/8eb73c91-ae95-4d53-91bf-27ae02c854ce-kube-api-access-fd25t\") pod \"maas-api-revoke-sub-a1a24258-98rwv\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.308190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.308052    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a24258-98rwv\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.308190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.308098    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a24258-98rwv\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.308680 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.308661    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a24258-98rwv\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.310397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.310372    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a24258-98rwv\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.320546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.320518    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd25t\" (UniqueName: \"kubernetes.io/projected/8eb73c91-ae95-4d53-91bf-27ae02c854ce-kube-api-access-fd25t\") pod \"maas-api-revoke-sub-a1a24258-98rwv\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.430644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.430563    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:31.724013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.723978    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"]
Nov 03 13:44:31.731417 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:44:31.731384    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb73c91_ae95_4d53_91bf_27ae02c854ce.slice/crio-ce9696c2f1637768927fb25984220fbb67e9e2c382b3f9b36d794d8b2d3430e6 WatchSource:0}: Error finding container ce9696c2f1637768927fb25984220fbb67e9e2c382b3f9b36d794d8b2d3430e6: Status 404 returned error can't find the container with id ce9696c2f1637768927fb25984220fbb67e9e2c382b3f9b36d794d8b2d3430e6
Nov 03 13:44:31.750331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:31.750307    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:32.151559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:32.151513    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv" event={"ID":"8eb73c91-ae95-4d53-91bf-27ae02c854ce","Type":"ContainerStarted","Data":"65f211f08b8166b8c1f25c2847da0161b33c7b0e3d9abd6d9bd0202ad24854f4"}
Nov 03 13:44:32.151724 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:32.151579    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv" event={"ID":"8eb73c91-ae95-4d53-91bf-27ae02c854ce","Type":"ContainerStarted","Data":"ce9696c2f1637768927fb25984220fbb67e9e2c382b3f9b36d794d8b2d3430e6"}
Nov 03 13:44:32.198837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:32.198791    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv" podStartSLOduration=1.198777077 podStartE2EDuration="1.198777077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:44:32.197699896 +0000 UTC m=+2113.275464923" watchObservedRunningTime="2026-11-03 13:44:32.198777077 +0000 UTC m=+2113.276542103"
Nov 03 13:44:32.208882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:32.208858    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"]
Nov 03 13:44:32.218193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:32.218166    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"]
Nov 03 13:44:32.349844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:32.349764    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:32.538094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:32.538060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:32.709404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:32.709372    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:33.159201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:33.159169    5420 generic.go:356] "Generic (PLEG): container finished" podID="8eb73c91-ae95-4d53-91bf-27ae02c854ce" containerID="65f211f08b8166b8c1f25c2847da0161b33c7b0e3d9abd6d9bd0202ad24854f4" exitCode=0
Nov 03 13:44:33.159614 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:33.159259    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv" event={"ID":"8eb73c91-ae95-4d53-91bf-27ae02c854ce","Type":"ContainerDied","Data":"65f211f08b8166b8c1f25c2847da0161b33c7b0e3d9abd6d9bd0202ad24854f4"}
Nov 03 13:44:33.198088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:33.198062    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:33.514021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:33.513992    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:34.233640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.233604    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:34.310025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.310005    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:34.446268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.446163    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb73c91-ae95-4d53-91bf-27ae02c854ce-kube-api-access-fd25t\" (UniqueName: \"kubernetes.io/projected/8eb73c91-ae95-4d53-91bf-27ae02c854ce-kube-api-access-fd25t\") pod \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") "
Nov 03 13:44:34.446268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.446202    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-openshift-ca\") pod \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") "
Nov 03 13:44:34.446489 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.446301    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-xks-ca\") pod \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\" (UID: \"8eb73c91-ae95-4d53-91bf-27ae02c854ce\") "
Nov 03 13:44:34.446591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.446567    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-openshift-ca" pod "8eb73c91-ae95-4d53-91bf-27ae02c854ce" (UID: "8eb73c91-ae95-4d53-91bf-27ae02c854ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:34.446686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.446672    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:34.448393 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.448366    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-xks-ca" pod "8eb73c91-ae95-4d53-91bf-27ae02c854ce" (UID: "8eb73c91-ae95-4d53-91bf-27ae02c854ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:34.448550 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.448432    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb73c91-ae95-4d53-91bf-27ae02c854ce-kube-api-access-fd25t" pod "8eb73c91-ae95-4d53-91bf-27ae02c854ce" (UID: "8eb73c91-ae95-4d53-91bf-27ae02c854ce"). InnerVolumeSpecName "kube-api-access-fd25t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:34.547893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.547864    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb73c91-ae95-4d53-91bf-27ae02c854ce-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:34.547893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:34.547892    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd25t\" (UniqueName: \"kubernetes.io/projected/8eb73c91-ae95-4d53-91bf-27ae02c854ce-kube-api-access-fd25t\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:35.126257 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:35.126227    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:35.172188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:35.172154    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv" event={"ID":"8eb73c91-ae95-4d53-91bf-27ae02c854ce","Type":"ContainerDied","Data":"ce9696c2f1637768927fb25984220fbb67e9e2c382b3f9b36d794d8b2d3430e6"}
Nov 03 13:44:35.172349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:35.172199    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce9696c2f1637768927fb25984220fbb67e9e2c382b3f9b36d794d8b2d3430e6"
Nov 03 13:44:35.172349 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:35.172166    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"
Nov 03 13:44:35.722174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:35.722147    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:35.927569 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:35.927539    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:36.226875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:36.226847    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:36.652762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:36.652688    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:37.209951 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:37.209914    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp" podUID="7149f288-00a3-40a7-9761-590f94057b00" containerName="payload-processing" containerID="cri-o://3f8c4d29ab1ba7255b17cbf4d974bd461113acc4021a7738e1a4a24301096533" gracePeriod=25
Nov 03 13:44:37.219434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:37.219373    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd" podUID="b788ffb7-d4b9-4ed0-a537-48ab5f115814" containerName="payload-pre-processing" containerID="cri-o://6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43" gracePeriod=25
Nov 03 13:44:37.228369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:37.228348    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:37.522762 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:37.522682    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:38.335008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:38.334973    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:39.017519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:39.017488    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:39.736798 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:39.736765    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:39.768793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:39.768765    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:44:39.898104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:39.898072    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:40.097938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:40.097847    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:40.427677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:40.427531    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:40.690843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:40.690769    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"]
Nov 03 13:44:40.691105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:40.691067    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy" containerID="cri-o://7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d" gracePeriod=30
Nov 03 13:44:40.733628 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:40.733595    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:40.911677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:40.911642    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:41.464613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.464583    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:41.656418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.656379    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:41.658074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.658043    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"]
Nov 03 13:44:41.658595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.658577    5420 cpu_manager.go:414] "containerMap: removing container" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy"
Nov 03 13:44:41.658681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.658597    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52845948-0621-4d3a-895d-b57495762003" containerName="istio-proxy"
Nov 03 13:44:41.658681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.658675    5420 cpu_manager.go:414] "containerMap: removing container" podUID="8eb73c91-ae95-4d53-91bf-27ae02c854ce" containerName="revoke-keys"
Nov 03 13:44:41.658793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.658685    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb73c91-ae95-4d53-91bf-27ae02c854ce" containerName="revoke-keys"
Nov 03 13:44:41.658793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.658787    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb73c91-ae95-4d53-91bf-27ae02c854ce" containerName="revoke-keys"
Nov 03 13:44:41.662341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.662324    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.669960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.669936    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:44:41.670113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.669992    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:44:41.670113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.670086    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:44:41.670267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.670179    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:44:41.674974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.674950    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"]
Nov 03 13:44:41.726036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.725973    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvwkm\" (UniqueName: \"kubernetes.io/projected/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-kube-api-access-kvwkm\") pod \"maas-api-revoke-sub-10ca9ee2-xpt5s\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.726036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.726023    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ca9ee2-xpt5s\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.726250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.726230    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ca9ee2-xpt5s\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.827455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.827428    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ca9ee2-xpt5s\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.827586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.827492    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvwkm\" (UniqueName: \"kubernetes.io/projected/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-kube-api-access-kvwkm\") pod \"maas-api-revoke-sub-10ca9ee2-xpt5s\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.827586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.827533    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ca9ee2-xpt5s\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.828079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.828054    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ca9ee2-xpt5s\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.829938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.829918    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ca9ee2-xpt5s\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.836777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.836755    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvwkm\" (UniqueName: \"kubernetes.io/projected/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-kube-api-access-kvwkm\") pod \"maas-api-revoke-sub-10ca9ee2-xpt5s\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:41.979341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:41.979272    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:42.272411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:42.272384    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"]
Nov 03 13:44:42.273938 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:44:42.273914    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7bf45b_a4e4_4a1a_b6f5_fafec399b6fb.slice/crio-8ad056f75b64b8003298aa0193e26badf62fe86be3e42e85ec32362a04889e85 WatchSource:0}: Error finding container 8ad056f75b64b8003298aa0193e26badf62fe86be3e42e85ec32362a04889e85: Status 404 returned error can't find the container with id 8ad056f75b64b8003298aa0193e26badf62fe86be3e42e85ec32362a04889e85
Nov 03 13:44:42.455756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:42.455725    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:43.229976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:43.229906    5420 generic.go:356] "Generic (PLEG): container finished" podID="eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb" containerID="93da930c9b91fe163668fe1be30ff5f91d6a9e15bd4e808608c95eaf1b5d9986" exitCode=0
Nov 03 13:44:43.231876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:43.230053    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s" event={"ID":"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb","Type":"ContainerDied","Data":"93da930c9b91fe163668fe1be30ff5f91d6a9e15bd4e808608c95eaf1b5d9986"}
Nov 03 13:44:43.231876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:43.230113    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s" event={"ID":"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb","Type":"ContainerStarted","Data":"8ad056f75b64b8003298aa0193e26badf62fe86be3e42e85ec32362a04889e85"}
Nov 03 13:44:43.274859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:43.274823    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:43.458863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:43.458833    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:43.658318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:43.658276    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:44.289200 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.289164    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:44.379364 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.379342    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:44.440413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.440381    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:44.457596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.457569    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-kube-api-access-kvwkm\" (UniqueName: \"kubernetes.io/projected/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-kube-api-access-kvwkm\") pod \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") "
Nov 03 13:44:44.457729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.457604    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-xks-ca\") pod \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") "
Nov 03 13:44:44.457729 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.457640    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-openshift-ca\") pod \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\" (UID: \"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb\") "
Nov 03 13:44:44.458063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.458040    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-openshift-ca" pod "eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb" (UID: "eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:44.459589 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.459561    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-kube-api-access-kvwkm" pod "eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb" (UID: "eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb"). InnerVolumeSpecName "kube-api-access-kvwkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:44.459738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.459717    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-xks-ca" pod "eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb" (UID: "eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:44:44.558576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.558492    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvwkm\" (UniqueName: \"kubernetes.io/projected/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-kube-api-access-kvwkm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:44.558576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.558521    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:44.558576 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:44.558531    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:45.062957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:45.062912    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:45.245977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:45.245947    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s" event={"ID":"eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb","Type":"ContainerDied","Data":"8ad056f75b64b8003298aa0193e26badf62fe86be3e42e85ec32362a04889e85"}
Nov 03 13:44:45.245977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:45.245964    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"
Nov 03 13:44:45.246173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:45.245997    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ad056f75b64b8003298aa0193e26badf62fe86be3e42e85ec32362a04889e85"
Nov 03 13:44:45.764563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:45.764526    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:45.787829 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:45.787789    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd" podUID="b788ffb7-d4b9-4ed0-a537-48ab5f115814" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:44:45.955279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:45.955255    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:44:46.074443 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074353    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-credential-socket\") pod \"85fbd726-c361-4e4e-8573-f92faefe6de0\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") "
Nov 03 13:44:46.074616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074463    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-certs\") pod \"85fbd726-c361-4e4e-8573-f92faefe6de0\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") "
Nov 03 13:44:46.074616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074495    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-podinfo\") pod \"85fbd726-c361-4e4e-8573-f92faefe6de0\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") "
Nov 03 13:44:46.074616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074526    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85fbd726-c361-4e4e-8573-f92faefe6de0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fbd726-c361-4e4e-8573-f92faefe6de0-istiod-ca-cert\") pod \"85fbd726-c361-4e4e-8573-f92faefe6de0\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") "
Nov 03 13:44:46.074616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074578    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-data\") pod \"85fbd726-c361-4e4e-8573-f92faefe6de0\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") "
Nov 03 13:44:46.074616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074608    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-token\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-token\") pod \"85fbd726-c361-4e4e-8573-f92faefe6de0\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") "
Nov 03 13:44:46.074934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074660    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-kube-api-access-q59rh\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-kube-api-access-q59rh\") pod \"85fbd726-c361-4e4e-8573-f92faefe6de0\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") "
Nov 03 13:44:46.074934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074657    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-credential-socket" pod "85fbd726-c361-4e4e-8573-f92faefe6de0" (UID: "85fbd726-c361-4e4e-8573-f92faefe6de0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.074934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074777    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-socket\") pod \"85fbd726-c361-4e4e-8573-f92faefe6de0\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") "
Nov 03 13:44:46.074934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074833    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-envoy\") pod \"85fbd726-c361-4e4e-8573-f92faefe6de0\" (UID: \"85fbd726-c361-4e4e-8573-f92faefe6de0\") "
Nov 03 13:44:46.074934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.074774    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-certs" pod "85fbd726-c361-4e4e-8573-f92faefe6de0" (UID: "85fbd726-c361-4e4e-8573-f92faefe6de0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.075192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.075030    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-socket" pod "85fbd726-c361-4e4e-8573-f92faefe6de0" (UID: "85fbd726-c361-4e4e-8573-f92faefe6de0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.075192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.075077    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85fbd726-c361-4e4e-8573-f92faefe6de0-istiod-ca-cert" pod "85fbd726-c361-4e4e-8573-f92faefe6de0" (UID: "85fbd726-c361-4e4e-8573-f92faefe6de0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:46.075368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.075224    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-data" pod "85fbd726-c361-4e4e-8573-f92faefe6de0" (UID: "85fbd726-c361-4e4e-8573-f92faefe6de0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.075368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.075354    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.075493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.075371    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.075493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.075381    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.075493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.075390    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85fbd726-c361-4e4e-8573-f92faefe6de0-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.075493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.075399    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.076820 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.076797    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-token" pod "85fbd726-c361-4e4e-8573-f92faefe6de0" (UID: "85fbd726-c361-4e4e-8573-f92faefe6de0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:46.076925 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.076892    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-kube-api-access-q59rh" pod "85fbd726-c361-4e4e-8573-f92faefe6de0" (UID: "85fbd726-c361-4e4e-8573-f92faefe6de0"). InnerVolumeSpecName "kube-api-access-q59rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:46.077204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.077179    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-podinfo" pod "85fbd726-c361-4e4e-8573-f92faefe6de0" (UID: "85fbd726-c361-4e4e-8573-f92faefe6de0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:44:46.077204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.077180    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-envoy" pod "85fbd726-c361-4e4e-8573-f92faefe6de0" (UID: "85fbd726-c361-4e4e-8573-f92faefe6de0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:44:46.176593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.176561    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.176593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.176588    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.176593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.176600    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q59rh\" (UniqueName: \"kubernetes.io/projected/85fbd726-c361-4e4e-8573-f92faefe6de0-kube-api-access-q59rh\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.176835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.176609    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85fbd726-c361-4e4e-8573-f92faefe6de0-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:46.253970 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.253934    5420 generic.go:356] "Generic (PLEG): container finished" podID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerID="7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d" exitCode=0
Nov 03 13:44:46.254173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.253993    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"
Nov 03 13:44:46.254173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.254032    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" event={"ID":"85fbd726-c361-4e4e-8573-f92faefe6de0","Type":"ContainerDied","Data":"7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d"}
Nov 03 13:44:46.254173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.254102    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx" event={"ID":"85fbd726-c361-4e4e-8573-f92faefe6de0","Type":"ContainerDied","Data":"cefbfabd82d481e82892b517a6de854e7d754445933b910a66ffa8a72d19e4fd"}
Nov 03 13:44:46.254173 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.254131    5420 scope.go:122] "RemoveContainer" containerID="7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d"
Nov 03 13:44:46.264439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.264418    5420 scope.go:122] "RemoveContainer" containerID="7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d"
Nov 03 13:44:46.264702 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:46.264685    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d\": container with ID starting with 7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d not found: ID does not exist" containerID="7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d"
Nov 03 13:44:46.264757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.264709    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d"} err="failed to get container status \"7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d\": rpc error: code = NotFound desc = could not find container \"7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d\": container with ID starting with 7977c5e92af8ea7a5e18e70773afae20ffb4cfe84754c8ddc814707906e1326d not found: ID does not exist"
Nov 03 13:44:46.297949 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.297920    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"]
Nov 03 13:44:46.305621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.305594    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-jzdqx"]
Nov 03 13:44:46.466998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.466953    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:46.797813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:46.797731    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp" podUID="7149f288-00a3-40a7-9761-590f94057b00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:44:47.085076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:47.084998    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" path="/var/lib/kubelet/pods/85fbd726-c361-4e4e-8573-f92faefe6de0/volumes"
Nov 03 13:44:47.293144 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:47.293081    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:48.025165 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:48.025109    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:48.767904 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:48.767866    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:49.794525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:49.794485    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:50.407775 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:50.407723    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:50.607702 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:50.607631    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:50.956448 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:50.956421    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:51.159665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:51.159630    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:51.685556 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:51.685520    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:52.303488 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:52.303451    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:52.497802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:52.497768    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:52.985409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:52.985381    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:53.757167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:53.757136    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:54.421865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:54.421837    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:54.731008 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:54.730977    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:44:55.120274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:55.120163    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:55.499196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:55.499151    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:44:55.787587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:55.787494    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd" podUID="b788ffb7-d4b9-4ed0-a537-48ab5f115814" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:44:55.808013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:55.807968    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:56.690555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:56.690483    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:56.797708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:56.797680    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp" podUID="7149f288-00a3-40a7-9761-590f94057b00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:44:57.310771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.310740    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:57.330891 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.330863    5420 generic.go:356] "Generic (PLEG): container finished" podID="7149f288-00a3-40a7-9761-590f94057b00" containerID="3f8c4d29ab1ba7255b17cbf4d974bd461113acc4021a7738e1a4a24301096533" exitCode=0
Nov 03 13:44:57.331020 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.330933    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp" event={"ID":"7149f288-00a3-40a7-9761-590f94057b00","Type":"ContainerDied","Data":"3f8c4d29ab1ba7255b17cbf4d974bd461113acc4021a7738e1a4a24301096533"}
Nov 03 13:44:57.480611 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.480553    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:44:57.505032 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.501195    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:44:57.594427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.594399    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b788ffb7-d4b9-4ed0-a537-48ab5f115814-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b788ffb7-d4b9-4ed0-a537-48ab5f115814-plugins-config-volume\") pod \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\" (UID: \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\") "
Nov 03 13:44:57.594595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.594470    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7149f288-00a3-40a7-9761-590f94057b00-kube-api-access-t4lkk\" (UniqueName: \"kubernetes.io/projected/7149f288-00a3-40a7-9761-590f94057b00-kube-api-access-t4lkk\") pod \"7149f288-00a3-40a7-9761-590f94057b00\" (UID: \"7149f288-00a3-40a7-9761-590f94057b00\") "
Nov 03 13:44:57.594595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.594582    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7149f288-00a3-40a7-9761-590f94057b00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149f288-00a3-40a7-9761-590f94057b00-plugins-config-volume\") pod \"7149f288-00a3-40a7-9761-590f94057b00\" (UID: \"7149f288-00a3-40a7-9761-590f94057b00\") "
Nov 03 13:44:57.594677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.594612    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b788ffb7-d4b9-4ed0-a537-48ab5f115814-kube-api-access-s5cjq\" (UniqueName: \"kubernetes.io/projected/b788ffb7-d4b9-4ed0-a537-48ab5f115814-kube-api-access-s5cjq\") pod \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\" (UID: \"b788ffb7-d4b9-4ed0-a537-48ab5f115814\") "
Nov 03 13:44:57.594967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.594929    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b788ffb7-d4b9-4ed0-a537-48ab5f115814-plugins-config-volume" pod "b788ffb7-d4b9-4ed0-a537-48ab5f115814" (UID: "b788ffb7-d4b9-4ed0-a537-48ab5f115814"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:57.595066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.594960    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7149f288-00a3-40a7-9761-590f94057b00-plugins-config-volume" pod "7149f288-00a3-40a7-9761-590f94057b00" (UID: "7149f288-00a3-40a7-9761-590f94057b00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:44:57.595066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.595057    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149f288-00a3-40a7-9761-590f94057b00-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:57.595135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.595076    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b788ffb7-d4b9-4ed0-a537-48ab5f115814-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:57.596612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.596582    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7149f288-00a3-40a7-9761-590f94057b00-kube-api-access-t4lkk" pod "7149f288-00a3-40a7-9761-590f94057b00" (UID: "7149f288-00a3-40a7-9761-590f94057b00"). InnerVolumeSpecName "kube-api-access-t4lkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:57.596782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.596763    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b788ffb7-d4b9-4ed0-a537-48ab5f115814-kube-api-access-s5cjq" pod "b788ffb7-d4b9-4ed0-a537-48ab5f115814" (UID: "b788ffb7-d4b9-4ed0-a537-48ab5f115814"). InnerVolumeSpecName "kube-api-access-s5cjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:44:57.696311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.696278    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5cjq\" (UniqueName: \"kubernetes.io/projected/b788ffb7-d4b9-4ed0-a537-48ab5f115814-kube-api-access-s5cjq\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:57.696311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:57.696308    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4lkk\" (UniqueName: \"kubernetes.io/projected/7149f288-00a3-40a7-9761-590f94057b00-kube-api-access-t4lkk\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:44:58.174977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.174947    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:58.338033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.338008    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"
Nov 03 13:44:58.338187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.338029    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp" event={"ID":"7149f288-00a3-40a7-9761-590f94057b00","Type":"ContainerDied","Data":"39bb0cefa84c9efcf5d7614e53c5fcc6c7c7ac6dc8d62016953e1887244b870f"}
Nov 03 13:44:58.338187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.338088    5420 scope.go:122] "RemoveContainer" containerID="3f8c4d29ab1ba7255b17cbf4d974bd461113acc4021a7738e1a4a24301096533"
Nov 03 13:44:58.343612 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.343587    5420 generic.go:356] "Generic (PLEG): container finished" podID="b788ffb7-d4b9-4ed0-a537-48ab5f115814" containerID="6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43" exitCode=0
Nov 03 13:44:58.343739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.343652    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"
Nov 03 13:44:58.343739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.343659    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd" event={"ID":"b788ffb7-d4b9-4ed0-a537-48ab5f115814","Type":"ContainerDied","Data":"6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43"}
Nov 03 13:44:58.343739 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.343701    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd" event={"ID":"b788ffb7-d4b9-4ed0-a537-48ab5f115814","Type":"ContainerDied","Data":"e815ae99071ad9dc1d815eaa183ce878f55287f1ff5ea09c8b8e3206085dbafd"}
Nov 03 13:44:58.349779 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.349761    5420 scope.go:122] "RemoveContainer" containerID="6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43"
Nov 03 13:44:58.358727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.358706    5420 scope.go:122] "RemoveContainer" containerID="6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43"
Nov 03 13:44:58.358975 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:44:58.358957    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43\": container with ID starting with 6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43 not found: ID does not exist" containerID="6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43"
Nov 03 13:44:58.359024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.358983    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43"} err="failed to get container status \"6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43\": rpc error: code = NotFound desc = could not find container \"6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43\": container with ID starting with 6419af22ce591ebb1404701ae7a359f8d05b8e99040e0e1ec88b8e889324be43 not found: ID does not exist"
Nov 03 13:44:58.371237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.371197    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"]
Nov 03 13:44:58.377444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.377423    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8f8aa8-5886b66df7-x64mp"]
Nov 03 13:44:58.394192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.394172    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"]
Nov 03 13:44:58.396918 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:58.396892    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8f8aa8-bf5f89f4c-9k7qd"]
Nov 03 13:44:59.012583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:59.012553    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:59.083579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:59.083549    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7149f288-00a3-40a7-9761-590f94057b00" path="/var/lib/kubelet/pods/7149f288-00a3-40a7-9761-590f94057b00/volumes"
Nov 03 13:44:59.083931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:59.083919    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b788ffb7-d4b9-4ed0-a537-48ab5f115814" path="/var/lib/kubelet/pods/b788ffb7-d4b9-4ed0-a537-48ab5f115814/volumes"
Nov 03 13:44:59.197624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:59.197600    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:59.543508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:59.543477    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:59.785367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:59.785339    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:59.985763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:44:59.985734    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:00.164868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.164835    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"]
Nov 03 13:45:00.165290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165277    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7149f288-00a3-40a7-9761-590f94057b00" containerName="payload-processing"
Nov 03 13:45:00.165290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165290    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7149f288-00a3-40a7-9761-590f94057b00" containerName="payload-processing"
Nov 03 13:45:00.165391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165300    5420 cpu_manager.go:414] "containerMap: removing container" podUID="eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb" containerName="revoke-keys"
Nov 03 13:45:00.165391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165305    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb" containerName="revoke-keys"
Nov 03 13:45:00.165391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165326    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b788ffb7-d4b9-4ed0-a537-48ab5f115814" containerName="payload-pre-processing"
Nov 03 13:45:00.165391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165333    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b788ffb7-d4b9-4ed0-a537-48ab5f115814" containerName="payload-pre-processing"
Nov 03 13:45:00.165391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165342    5420 cpu_manager.go:414] "containerMap: removing container" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy"
Nov 03 13:45:00.165391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165347    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy"
Nov 03 13:45:00.165616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165434    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b788ffb7-d4b9-4ed0-a537-48ab5f115814" containerName="payload-pre-processing"
Nov 03 13:45:00.165616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165444    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="85fbd726-c361-4e4e-8573-f92faefe6de0" containerName="istio-proxy"
Nov 03 13:45:00.165616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165453    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7149f288-00a3-40a7-9761-590f94057b00" containerName="payload-processing"
Nov 03 13:45:00.165616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.165459    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb" containerName="revoke-keys"
Nov 03 13:45:00.170175 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.170154    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:00.174763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.174739    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"]
Nov 03 13:45:00.179009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.178987    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"]
Nov 03 13:45:00.179125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.179101    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:00.182476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.182456    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:00.183309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.183295    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:45:00.186445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.186430    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:45:00.187677 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.187657    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:45:00.191678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.191653    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"]
Nov 03 13:45:00.195201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.195159    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"]
Nov 03 13:45:00.219976 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.219952    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"]
Nov 03 13:45:00.236232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.236165    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:00.323573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.323545    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc\" (UID: \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:00.323706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.323608    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plllz\" (UniqueName: \"kubernetes.io/projected/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-kube-api-access-plllz\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc\" (UID: \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:00.323706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.323671    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81842164-51e8-4b5b-b59d-1e5a50fc94ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw\" (UID: \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:00.323812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.323731    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9\" (UID: \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:00.323812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.323794    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlnm\" (UniqueName: \"kubernetes.io/projected/81842164-51e8-4b5b-b59d-1e5a50fc94ab-kube-api-access-5xlnm\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw\" (UID: \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:00.323884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.323834    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5hp\" (UniqueName: \"kubernetes.io/projected/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-kube-api-access-tw5hp\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9\" (UID: \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:00.424840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.424805    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5hp\" (UniqueName: \"kubernetes.io/projected/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-kube-api-access-tw5hp\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9\" (UID: \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:00.425043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.424855    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc\" (UID: \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:00.425043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.424898    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plllz\" (UniqueName: \"kubernetes.io/projected/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-kube-api-access-plllz\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc\" (UID: \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:00.425043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.424925    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81842164-51e8-4b5b-b59d-1e5a50fc94ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw\" (UID: \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:00.425043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.424982    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9\" (UID: \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:00.425043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.425001    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlnm\" (UniqueName: \"kubernetes.io/projected/81842164-51e8-4b5b-b59d-1e5a50fc94ab-kube-api-access-5xlnm\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw\" (UID: \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:00.425716 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.425697    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9\" (UID: \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:00.425776 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.425754    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81842164-51e8-4b5b-b59d-1e5a50fc94ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw\" (UID: \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:00.425811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.425754    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc\" (UID: \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:00.436339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.436313    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plllz\" (UniqueName: \"kubernetes.io/projected/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-kube-api-access-plllz\") pod \"maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc\" (UID: \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:00.436476 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.436435    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5hp\" (UniqueName: \"kubernetes.io/projected/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-kube-api-access-tw5hp\") pod \"maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9\" (UID: \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:00.439077 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.439057    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlnm\" (UniqueName: \"kubernetes.io/projected/81842164-51e8-4b5b-b59d-1e5a50fc94ab-kube-api-access-5xlnm\") pod \"maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw\" (UID: \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:00.448977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.448953    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:00.484459 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.484435    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:00.495609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.495581    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:00.507588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.507562    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:00.684922 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.684895    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:00.789585 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.789543    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"]
Nov 03 13:45:00.789808 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:45:00.789781    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cdaa0a_1931_4f1e_bb21_aaa659790c1b.slice/crio-1807e6c27bdc1c84d23af2f693311a5837fe310ea55db102a82979904d93adaa WatchSource:0}: Error finding container 1807e6c27bdc1c84d23af2f693311a5837fe310ea55db102a82979904d93adaa: Status 404 returned error can't find the container with id 1807e6c27bdc1c84d23af2f693311a5837fe310ea55db102a82979904d93adaa
Nov 03 13:45:00.819433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.819409    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"]
Nov 03 13:45:00.819535 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:45:00.819442    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690c206f_16a1_4f30_a7fb_0c2f9dd15ef3.slice/crio-52c09d5eff84f296f2d10b6367fa5a199a9909d41144dd5dc015b926c8a692d4 WatchSource:0}: Error finding container 52c09d5eff84f296f2d10b6367fa5a199a9909d41144dd5dc015b926c8a692d4: Status 404 returned error can't find the container with id 52c09d5eff84f296f2d10b6367fa5a199a9909d41144dd5dc015b926c8a692d4
Nov 03 13:45:00.828707 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:00.828688    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"]
Nov 03 13:45:00.830012 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:45:00.829991    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81842164_51e8_4b5b_b59d_1e5a50fc94ab.slice/crio-b6370ab7b036c964cd33dc11f2f3915d8e051c0427636139c2c6f6e600a5eda4 WatchSource:0}: Error finding container b6370ab7b036c964cd33dc11f2f3915d8e051c0427636139c2c6f6e600a5eda4: Status 404 returned error can't find the container with id b6370ab7b036c964cd33dc11f2f3915d8e051c0427636139c2c6f6e600a5eda4
Nov 03 13:45:01.338652 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.338567    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:01.374152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.374123    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" event={"ID":"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3","Type":"ContainerStarted","Data":"4570a0d1e16f503b7e37e94e2b8db0191b9059270e28c8c4949ab23c1e72d8a0"}
Nov 03 13:45:01.374319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.374179    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" event={"ID":"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3","Type":"ContainerStarted","Data":"52c09d5eff84f296f2d10b6367fa5a199a9909d41144dd5dc015b926c8a692d4"}
Nov 03 13:45:01.375827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.375803    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" event={"ID":"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b","Type":"ContainerStarted","Data":"f0bc4f89092cac414bdc3193fef159009ee4a4b8413cc542f62bf5ef23b23a37"}
Nov 03 13:45:01.375975 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.375845    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" event={"ID":"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b","Type":"ContainerStarted","Data":"1807e6c27bdc1c84d23af2f693311a5837fe310ea55db102a82979904d93adaa"}
Nov 03 13:45:01.377377 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.377351    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" event={"ID":"81842164-51e8-4b5b-b59d-1e5a50fc94ab","Type":"ContainerStarted","Data":"fddb78269363322b512a084ce959588001b01f8779e4326af4bc808e588df48a"}
Nov 03 13:45:01.377477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.377392    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" event={"ID":"81842164-51e8-4b5b-b59d-1e5a50fc94ab","Type":"ContainerStarted","Data":"b6370ab7b036c964cd33dc11f2f3915d8e051c0427636139c2c6f6e600a5eda4"}
Nov 03 13:45:01.420297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.420240    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" podStartSLOduration=1.420228599 podStartE2EDuration="1.420228599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:01.417916243 +0000 UTC m=+2142.495681270" watchObservedRunningTime="2026-11-03 13:45:01.420228599 +0000 UTC m=+2142.497993619"
Nov 03 13:45:01.455306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.455266    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" podStartSLOduration=1.4552531819999999 podStartE2EDuration="1.455253182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:01.453417181 +0000 UTC m=+2142.531182207" watchObservedRunningTime="2026-11-03 13:45:01.455253182 +0000 UTC m=+2142.533018208"
Nov 03 13:45:01.482116 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:01.482074    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" podStartSLOduration=1.482059839 podStartE2EDuration="1.482059839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:01.480901131 +0000 UTC m=+2142.558666171" watchObservedRunningTime="2026-11-03 13:45:01.482059839 +0000 UTC m=+2142.559824866"
Nov 03 13:45:01.734936 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:01.734891    5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690c206f_16a1_4f30_a7fb_0c2f9dd15ef3.slice/crio-conmon-4570a0d1e16f503b7e37e94e2b8db0191b9059270e28c8c4949ab23c1e72d8a0.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 13:45:02.289557 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.289517    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:02.385573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.385540    5420 generic.go:356] "Generic (PLEG): container finished" podID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerID="f0bc4f89092cac414bdc3193fef159009ee4a4b8413cc542f62bf5ef23b23a37" exitCode=60
Nov 03 13:45:02.386041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.385624    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" event={"ID":"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b","Type":"ContainerDied","Data":"f0bc4f89092cac414bdc3193fef159009ee4a4b8413cc542f62bf5ef23b23a37"}
Nov 03 13:45:02.386041 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.385849    5420 scope.go:122] "RemoveContainer" containerID="f0bc4f89092cac414bdc3193fef159009ee4a4b8413cc542f62bf5ef23b23a37"
Nov 03 13:45:02.387331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.387267    5420 generic.go:356] "Generic (PLEG): container finished" podID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerID="fddb78269363322b512a084ce959588001b01f8779e4326af4bc808e588df48a" exitCode=60
Nov 03 13:45:02.387413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.387346    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" event={"ID":"81842164-51e8-4b5b-b59d-1e5a50fc94ab","Type":"ContainerDied","Data":"fddb78269363322b512a084ce959588001b01f8779e4326af4bc808e588df48a"}
Nov 03 13:45:02.387579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.387563    5420 scope.go:122] "RemoveContainer" containerID="fddb78269363322b512a084ce959588001b01f8779e4326af4bc808e588df48a"
Nov 03 13:45:02.389117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.389098    5420 generic.go:356] "Generic (PLEG): container finished" podID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerID="4570a0d1e16f503b7e37e94e2b8db0191b9059270e28c8c4949ab23c1e72d8a0" exitCode=60
Nov 03 13:45:02.389187 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.389147    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" event={"ID":"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3","Type":"ContainerDied","Data":"4570a0d1e16f503b7e37e94e2b8db0191b9059270e28c8c4949ab23c1e72d8a0"}
Nov 03 13:45:02.389419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.389404    5420 scope.go:122] "RemoveContainer" containerID="4570a0d1e16f503b7e37e94e2b8db0191b9059270e28c8c4949ab23c1e72d8a0"
Nov 03 13:45:02.505049 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:02.505016    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:03.179784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.179748    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:03.401274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.401244    5420 generic.go:356] "Generic (PLEG): container finished" podID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerID="f2effca464abb5109807b99fa5e1e9a789b4fbcfff9ce2a92503f34ac8e13908" exitCode=60
Nov 03 13:45:03.401687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.401320    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" event={"ID":"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b","Type":"ContainerDied","Data":"f2effca464abb5109807b99fa5e1e9a789b4fbcfff9ce2a92503f34ac8e13908"}
Nov 03 13:45:03.401687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.401393    5420 scope.go:122] "RemoveContainer" containerID="f0bc4f89092cac414bdc3193fef159009ee4a4b8413cc542f62bf5ef23b23a37"
Nov 03 13:45:03.401837 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.401688    5420 scope.go:122] "RemoveContainer" containerID="f2effca464abb5109807b99fa5e1e9a789b4fbcfff9ce2a92503f34ac8e13908"
Nov 03 13:45:03.401996 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:03.401965    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc_odh-ai-gateway-infra(b2cdaa0a-1931-4f1e-bb21-aaa659790c1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b"
Nov 03 13:45:03.403957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.403935    5420 generic.go:356] "Generic (PLEG): container finished" podID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerID="376fceab60c6b12799c4feea739bc7e45ff489290c0c936535a7b7efedcd4a0a" exitCode=60
Nov 03 13:45:03.404139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.404009    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" event={"ID":"81842164-51e8-4b5b-b59d-1e5a50fc94ab","Type":"ContainerDied","Data":"376fceab60c6b12799c4feea739bc7e45ff489290c0c936535a7b7efedcd4a0a"}
Nov 03 13:45:03.404297 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.404278    5420 scope.go:122] "RemoveContainer" containerID="376fceab60c6b12799c4feea739bc7e45ff489290c0c936535a7b7efedcd4a0a"
Nov 03 13:45:03.404535 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:03.404517    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw_odh-ai-gateway-infra(81842164-51e8-4b5b-b59d-1e5a50fc94ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab"
Nov 03 13:45:03.406029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.405999    5420 generic.go:356] "Generic (PLEG): container finished" podID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerID="202f7bf20b46f14e81da851ea831aa9bd8e11e1abb1f3b104189159a2657b199" exitCode=60
Nov 03 13:45:03.406134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.406029    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" event={"ID":"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3","Type":"ContainerDied","Data":"202f7bf20b46f14e81da851ea831aa9bd8e11e1abb1f3b104189159a2657b199"}
Nov 03 13:45:03.406341 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.406325    5420 scope.go:122] "RemoveContainer" containerID="202f7bf20b46f14e81da851ea831aa9bd8e11e1abb1f3b104189159a2657b199"
Nov 03 13:45:03.406579 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:03.406561    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9_odh-ai-gateway-infra(690c206f-16a1-4f30-a7fb-0c2f9dd15ef3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3"
Nov 03 13:45:03.417285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.417265    5420 scope.go:122] "RemoveContainer" containerID="fddb78269363322b512a084ce959588001b01f8779e4326af4bc808e588df48a"
Nov 03 13:45:03.429510 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:03.429493    5420 scope.go:122] "RemoveContainer" containerID="4570a0d1e16f503b7e37e94e2b8db0191b9059270e28c8c4949ab23c1e72d8a0"
Nov 03 13:45:04.291004 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:04.290966    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:04.414965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:04.414938    5420 scope.go:122] "RemoveContainer" containerID="202f7bf20b46f14e81da851ea831aa9bd8e11e1abb1f3b104189159a2657b199"
Nov 03 13:45:04.415450 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:04.415268    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9_odh-ai-gateway-infra(690c206f-16a1-4f30-a7fb-0c2f9dd15ef3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3"
Nov 03 13:45:04.416801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:04.416774    5420 scope.go:122] "RemoveContainer" containerID="f2effca464abb5109807b99fa5e1e9a789b4fbcfff9ce2a92503f34ac8e13908"
Nov 03 13:45:04.417018 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:04.417000    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc_odh-ai-gateway-infra(b2cdaa0a-1931-4f1e-bb21-aaa659790c1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b"
Nov 03 13:45:04.418254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:04.418238    5420 scope.go:122] "RemoveContainer" containerID="376fceab60c6b12799c4feea739bc7e45ff489290c0c936535a7b7efedcd4a0a"
Nov 03 13:45:04.418427 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:04.418411    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw_odh-ai-gateway-infra(81842164-51e8-4b5b-b59d-1e5a50fc94ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab"
Nov 03 13:45:05.090978 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:05.090940    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:05.395959 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:05.395877    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"]
Nov 03 13:45:05.409405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:05.409376    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"]
Nov 03 13:45:05.569085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:05.569050    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:06.187410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:06.187378    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:06.848338 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:06.848304    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:06.852749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:06.852721    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:45:07.672136 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:07.672108    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:08.510286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:08.510258    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:09.126619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:09.126587    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:09.446671 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:09.446547    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:10.280865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:10.280831    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:11.075911 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:11.075884    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:11.733990 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:11.733950    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:12.428922 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:12.428887    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:13.261546 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:13.261515    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:13.842069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:13.842037    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:14.003946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:14.003915    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:14.076870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:14.076841    5420 scope.go:122] "RemoveContainer" containerID="f2effca464abb5109807b99fa5e1e9a789b4fbcfff9ce2a92503f34ac8e13908"
Nov 03 13:45:14.246199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:14.246174    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:14.387621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:14.387551    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"]
Nov 03 13:45:14.388052 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:14.387770    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy" containerID="cri-o://f8cf5dd9168b8af2fc0a1e1c4f65d33a4c5c474d12be7d328f5a43e31be4083d" gracePeriod=30
Nov 03 13:45:14.420278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:14.420230    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:14.488552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:14.488507    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" event={"ID":"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b","Type":"ContainerStarted","Data":"9d2c22512fef5fe5e8a6f4e6d52a426b91d5d080a0b64807ef9c43745d9f8c16"}
Nov 03 13:45:14.595536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:14.595501    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:15.077289 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.077261    5420 scope.go:122] "RemoveContainer" containerID="202f7bf20b46f14e81da851ea831aa9bd8e11e1abb1f3b104189159a2657b199"
Nov 03 13:45:15.137085 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.137060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:15.378368 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.378284    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:15.497073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.497035    5420 generic.go:356] "Generic (PLEG): container finished" podID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerID="9d2c22512fef5fe5e8a6f4e6d52a426b91d5d080a0b64807ef9c43745d9f8c16" exitCode=60
Nov 03 13:45:15.497620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.497121    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" event={"ID":"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b","Type":"ContainerDied","Data":"9d2c22512fef5fe5e8a6f4e6d52a426b91d5d080a0b64807ef9c43745d9f8c16"}
Nov 03 13:45:15.497620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.497200    5420 scope.go:122] "RemoveContainer" containerID="f2effca464abb5109807b99fa5e1e9a789b4fbcfff9ce2a92503f34ac8e13908"
Nov 03 13:45:15.497620 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.497563    5420 scope.go:122] "RemoveContainer" containerID="9d2c22512fef5fe5e8a6f4e6d52a426b91d5d080a0b64807ef9c43745d9f8c16"
Nov 03 13:45:15.497853 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:15.497834    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc_odh-ai-gateway-infra(b2cdaa0a-1931-4f1e-bb21-aaa659790c1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b"
Nov 03 13:45:15.499999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.499974    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" event={"ID":"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3","Type":"ContainerStarted","Data":"a01d24b7a8feea88ffcbd0ad4c3e51749582c176c54b1983b47954546dcc251a"}
Nov 03 13:45:15.539125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.539098    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"]
Nov 03 13:45:15.991749 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:15.991695    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:16.099388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.099307    5420 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:45:16.099388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.099360    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:45:16.247419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.247390    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:16.510580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.510539    5420 generic.go:356] "Generic (PLEG): container finished" podID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerID="a01d24b7a8feea88ffcbd0ad4c3e51749582c176c54b1983b47954546dcc251a" exitCode=60
Nov 03 13:45:16.510963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.510608    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" event={"ID":"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3","Type":"ContainerDied","Data":"a01d24b7a8feea88ffcbd0ad4c3e51749582c176c54b1983b47954546dcc251a"}
Nov 03 13:45:16.510963 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.510685    5420 scope.go:122] "RemoveContainer" containerID="202f7bf20b46f14e81da851ea831aa9bd8e11e1abb1f3b104189159a2657b199"
Nov 03 13:45:16.511100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.511008    5420 scope.go:122] "RemoveContainer" containerID="a01d24b7a8feea88ffcbd0ad4c3e51749582c176c54b1983b47954546dcc251a"
Nov 03 13:45:16.511324 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:16.511303    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9_odh-ai-gateway-infra(690c206f-16a1-4f30-a7fb-0c2f9dd15ef3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3"
Nov 03 13:45:16.568708 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.568465    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"]
Nov 03 13:45:16.658848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.658825    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:16.691062 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.691032    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-maas-api-service-ca\") pod \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\" (UID: \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\") "
Nov 03 13:45:16.691253 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.691086    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-kube-api-access-plllz\" (UniqueName: \"kubernetes.io/projected/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-kube-api-access-plllz\") pod \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\" (UID: \"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b\") "
Nov 03 13:45:16.691493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.691467    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-maas-api-service-ca" pod "b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" (UID: "b2cdaa0a-1931-4f1e-bb21-aaa659790c1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:16.693281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.693255    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-kube-api-access-plllz" pod "b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" (UID: "b2cdaa0a-1931-4f1e-bb21-aaa659790c1b"). InnerVolumeSpecName "kube-api-access-plllz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:16.792623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.792548    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:16.792623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:16.792575    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plllz\" (UniqueName: \"kubernetes.io/projected/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b-kube-api-access-plllz\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:17.077009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.076931    5420 scope.go:122] "RemoveContainer" containerID="376fceab60c6b12799c4feea739bc7e45ff489290c0c936535a7b7efedcd4a0a"
Nov 03 13:45:17.105381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.105359    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:17.396910 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.396874    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:45:17.520147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.520120    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"
Nov 03 13:45:17.520586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.520148    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc" event={"ID":"b2cdaa0a-1931-4f1e-bb21-aaa659790c1b","Type":"ContainerDied","Data":"1807e6c27bdc1c84d23af2f693311a5837fe310ea55db102a82979904d93adaa"}
Nov 03 13:45:17.520586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.520245    5420 scope.go:122] "RemoveContainer" containerID="9d2c22512fef5fe5e8a6f4e6d52a426b91d5d080a0b64807ef9c43745d9f8c16"
Nov 03 13:45:17.523449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.522835    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" event={"ID":"81842164-51e8-4b5b-b59d-1e5a50fc94ab","Type":"ContainerStarted","Data":"f40c1bd658e25cf1a872f40b330852e8f67c2e671270786db1b52ffa8038e861"}
Nov 03 13:45:17.574328 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.574302    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"]
Nov 03 13:45:17.622284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.617158    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a4ab8e-29895225-ttgjc"]
Nov 03 13:45:17.677360 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.677334    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:17.710515 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.710480    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-maas-api-service-ca\") pod \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\" (UID: \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\") "
Nov 03 13:45:17.710689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.710596    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-kube-api-access-tw5hp\" (UniqueName: \"kubernetes.io/projected/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-kube-api-access-tw5hp\") pod \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\" (UID: \"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3\") "
Nov 03 13:45:17.710931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.710902    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-maas-api-service-ca" pod "690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" (UID: "690c206f-16a1-4f30-a7fb-0c2f9dd15ef3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:17.712914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.712883    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-kube-api-access-tw5hp" pod "690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" (UID: "690c206f-16a1-4f30-a7fb-0c2f9dd15ef3"). InnerVolumeSpecName "kube-api-access-tw5hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:17.811715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.811678    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw5hp\" (UniqueName: \"kubernetes.io/projected/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-kube-api-access-tw5hp\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:17.811715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:17.811707    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:18.251323 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.251293    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:18.532273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.532170    5420 generic.go:356] "Generic (PLEG): container finished" podID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerID="f40c1bd658e25cf1a872f40b330852e8f67c2e671270786db1b52ffa8038e861" exitCode=60
Nov 03 13:45:18.532273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.532246    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" event={"ID":"81842164-51e8-4b5b-b59d-1e5a50fc94ab","Type":"ContainerDied","Data":"f40c1bd658e25cf1a872f40b330852e8f67c2e671270786db1b52ffa8038e861"}
Nov 03 13:45:18.532734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.532312    5420 scope.go:122] "RemoveContainer" containerID="376fceab60c6b12799c4feea739bc7e45ff489290c0c936535a7b7efedcd4a0a"
Nov 03 13:45:18.532734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.532641    5420 scope.go:122] "RemoveContainer" containerID="f40c1bd658e25cf1a872f40b330852e8f67c2e671270786db1b52ffa8038e861"
Nov 03 13:45:18.532912 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:18.532887    5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw_odh-ai-gateway-infra(81842164-51e8-4b5b-b59d-1e5a50fc94ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab"
Nov 03 13:45:18.533723 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.533707    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"
Nov 03 13:45:18.533796 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.533728    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9" event={"ID":"690c206f-16a1-4f30-a7fb-0c2f9dd15ef3","Type":"ContainerDied","Data":"52c09d5eff84f296f2d10b6367fa5a199a9909d41144dd5dc015b926c8a692d4"}
Nov 03 13:45:18.544619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.544600    5420 scope.go:122] "RemoveContainer" containerID="a01d24b7a8feea88ffcbd0ad4c3e51749582c176c54b1983b47954546dcc251a"
Nov 03 13:45:18.567381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.567359    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"]
Nov 03 13:45:18.573892 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.573869    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7c192b-29895225-vqfr9"]
Nov 03 13:45:18.632479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.632453    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"]
Nov 03 13:45:18.956025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:18.955996    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:19.084131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.084100    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" path="/var/lib/kubelet/pods/690c206f-16a1-4f30-a7fb-0c2f9dd15ef3/volumes"
Nov 03 13:45:19.084566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.084550    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" path="/var/lib/kubelet/pods/b2cdaa0a-1931-4f1e-bb21-aaa659790c1b/volumes"
Nov 03 13:45:19.543964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.543931    5420 generic.go:356] "Generic (PLEG): container finished" podID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerID="f8cf5dd9168b8af2fc0a1e1c4f65d33a4c5c474d12be7d328f5a43e31be4083d" exitCode=0
Nov 03 13:45:19.544468 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.544005    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" event={"ID":"17f84e6b-bd58-4d15-a856-2900abe93ec4","Type":"ContainerDied","Data":"f8cf5dd9168b8af2fc0a1e1c4f65d33a4c5c474d12be7d328f5a43e31be4083d"}
Nov 03 13:45:19.636455 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.636363    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:19.654410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.654354    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:45:19.708809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.708773    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:19.731132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731097    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-credential-socket\") pod \"17f84e6b-bd58-4d15-a856-2900abe93ec4\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") "
Nov 03 13:45:19.731303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731147    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f84e6b-bd58-4d15-a856-2900abe93ec4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f84e6b-bd58-4d15-a856-2900abe93ec4-istiod-ca-cert\") pod \"17f84e6b-bd58-4d15-a856-2900abe93ec4\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") "
Nov 03 13:45:19.731303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731179    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-socket\") pod \"17f84e6b-bd58-4d15-a856-2900abe93ec4\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") "
Nov 03 13:45:19.731303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731235    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-certs\") pod \"17f84e6b-bd58-4d15-a856-2900abe93ec4\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") "
Nov 03 13:45:19.731303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731299    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-data\") pod \"17f84e6b-bd58-4d15-a856-2900abe93ec4\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") "
Nov 03 13:45:19.731533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731392    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-podinfo\") pod \"17f84e6b-bd58-4d15-a856-2900abe93ec4\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") "
Nov 03 13:45:19.731533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731423    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-credential-socket" pod "17f84e6b-bd58-4d15-a856-2900abe93ec4" (UID: "17f84e6b-bd58-4d15-a856-2900abe93ec4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:19.731533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731460    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-kube-api-access-kcc2d\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-kube-api-access-kcc2d\") pod \"17f84e6b-bd58-4d15-a856-2900abe93ec4\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") "
Nov 03 13:45:19.731533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731479    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-socket" pod "17f84e6b-bd58-4d15-a856-2900abe93ec4" (UID: "17f84e6b-bd58-4d15-a856-2900abe93ec4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:19.731533 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731524    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-token\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-token\") pod \"17f84e6b-bd58-4d15-a856-2900abe93ec4\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") "
Nov 03 13:45:19.731792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731574    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-envoy\") pod \"17f84e6b-bd58-4d15-a856-2900abe93ec4\" (UID: \"17f84e6b-bd58-4d15-a856-2900abe93ec4\") "
Nov 03 13:45:19.731792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731583    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-certs" pod "17f84e6b-bd58-4d15-a856-2900abe93ec4" (UID: "17f84e6b-bd58-4d15-a856-2900abe93ec4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:19.731953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731929    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-data" pod "17f84e6b-bd58-4d15-a856-2900abe93ec4" (UID: "17f84e6b-bd58-4d15-a856-2900abe93ec4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:19.731953 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731938    5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-credential-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.732069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731958    5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-socket\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.732069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.731974    5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-workload-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.732259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.732169    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f84e6b-bd58-4d15-a856-2900abe93ec4-istiod-ca-cert" pod "17f84e6b-bd58-4d15-a856-2900abe93ec4" (UID: "17f84e6b-bd58-4d15-a856-2900abe93ec4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:19.733633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.733612    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-token" pod "17f84e6b-bd58-4d15-a856-2900abe93ec4" (UID: "17f84e6b-bd58-4d15-a856-2900abe93ec4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:19.734009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.733986    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-podinfo" pod "17f84e6b-bd58-4d15-a856-2900abe93ec4" (UID: "17f84e6b-bd58-4d15-a856-2900abe93ec4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:45:19.734087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.733992    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-kube-api-access-kcc2d" pod "17f84e6b-bd58-4d15-a856-2900abe93ec4" (UID: "17f84e6b-bd58-4d15-a856-2900abe93ec4"). InnerVolumeSpecName "kube-api-access-kcc2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:19.734124 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.734106    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-envoy" pod "17f84e6b-bd58-4d15-a856-2900abe93ec4" (UID: "17f84e6b-bd58-4d15-a856-2900abe93ec4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:19.832565 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.832529    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81842164-51e8-4b5b-b59d-1e5a50fc94ab-kube-api-access-5xlnm\" (UniqueName: \"kubernetes.io/projected/81842164-51e8-4b5b-b59d-1e5a50fc94ab-kube-api-access-5xlnm\") pod \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\" (UID: \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\") "
Nov 03 13:45:19.832709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.832598    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81842164-51e8-4b5b-b59d-1e5a50fc94ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81842164-51e8-4b5b-b59d-1e5a50fc94ab-maas-api-service-ca\") pod \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\" (UID: \"81842164-51e8-4b5b-b59d-1e5a50fc94ab\") "
Nov 03 13:45:19.832936 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.832909    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81842164-51e8-4b5b-b59d-1e5a50fc94ab-maas-api-service-ca" pod "81842164-51e8-4b5b-b59d-1e5a50fc94ab" (UID: "81842164-51e8-4b5b-b59d-1e5a50fc94ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:19.833055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.832995    5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-podinfo\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.833055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.833006    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81842164-51e8-4b5b-b59d-1e5a50fc94ab-maas-api-service-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.833055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.833017    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcc2d\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-kube-api-access-kcc2d\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.833055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.833028    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.833055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.833036    5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-envoy\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.833055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.833044    5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f84e6b-bd58-4d15-a856-2900abe93ec4-istiod-ca-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.833055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.833052    5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f84e6b-bd58-4d15-a856-2900abe93ec4-istio-data\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:19.834388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.834362    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81842164-51e8-4b5b-b59d-1e5a50fc94ab-kube-api-access-5xlnm" pod "81842164-51e8-4b5b-b59d-1e5a50fc94ab" (UID: "81842164-51e8-4b5b-b59d-1e5a50fc94ab"). InnerVolumeSpecName "kube-api-access-5xlnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:19.882446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.882419    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:19.934042 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:19.933987    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xlnm\" (UniqueName: \"kubernetes.io/projected/81842164-51e8-4b5b-b59d-1e5a50fc94ab-kube-api-access-5xlnm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.416326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.416286    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main" containerID="cri-o://8c697e6621871ca1721c3a4d494a5f7fc6bcec34fdc5b5e86ef53c9685a586d6" gracePeriod=45
Nov 03 13:45:20.430673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.430644    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main" containerID="cri-o://8b93dc9d868d44062d10e5d2bb91229f923a55c36595551e5c516b1ea64e16ec" gracePeriod=45
Nov 03 13:45:20.455381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.455349    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:20.557383 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.557354    5420 generic.go:356] "Generic (PLEG): container finished" podID="fbbb9af9-1713-44f4-afcd-901667677352" containerID="8b93dc9d868d44062d10e5d2bb91229f923a55c36595551e5c516b1ea64e16ec" exitCode=0
Nov 03 13:45:20.557803 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.557446    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" event={"ID":"fbbb9af9-1713-44f4-afcd-901667677352","Type":"ContainerDied","Data":"8b93dc9d868d44062d10e5d2bb91229f923a55c36595551e5c516b1ea64e16ec"}
Nov 03 13:45:20.559169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.559139    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf" event={"ID":"17f84e6b-bd58-4d15-a856-2900abe93ec4","Type":"ContainerDied","Data":"b95d261f1db029214b76baae8c2d7f851aa77155a78db76b42076bb507e85e3b"}
Nov 03 13:45:20.559321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.559200    5420 scope.go:122] "RemoveContainer" containerID="f8cf5dd9168b8af2fc0a1e1c4f65d33a4c5c474d12be7d328f5a43e31be4083d"
Nov 03 13:45:20.559321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.559148    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"
Nov 03 13:45:20.561526 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.561506    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"
Nov 03 13:45:20.561665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.561554    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw" event={"ID":"81842164-51e8-4b5b-b59d-1e5a50fc94ab","Type":"ContainerDied","Data":"b6370ab7b036c964cd33dc11f2f3915d8e051c0427636139c2c6f6e600a5eda4"}
Nov 03 13:45:20.567280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.567253    5420 generic.go:356] "Generic (PLEG): container finished" podID="495bb85e-709f-405d-a92e-e681113bb0e1" containerID="8c697e6621871ca1721c3a4d494a5f7fc6bcec34fdc5b5e86ef53c9685a586d6" exitCode=0
Nov 03 13:45:20.567396 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.567368    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" event={"ID":"495bb85e-709f-405d-a92e-e681113bb0e1","Type":"ContainerDied","Data":"8c697e6621871ca1721c3a4d494a5f7fc6bcec34fdc5b5e86ef53c9685a586d6"}
Nov 03 13:45:20.577487 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.575463    5420 scope.go:122] "RemoveContainer" containerID="f40c1bd658e25cf1a872f40b330852e8f67c2e671270786db1b52ffa8038e861"
Nov 03 13:45:20.596139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.595847    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"]
Nov 03 13:45:20.598919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.598901    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-k4gjf"]
Nov 03 13:45:20.614737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.614591    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"]
Nov 03 13:45:20.622525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.622498    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ded926-29895225-dhrvw"]
Nov 03 13:45:20.685522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.685501    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:45:20.709835 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.709812    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:45:20.742385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.742347    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-home\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-home\") pod \"495bb85e-709f-405d-a92e-e681113bb0e1\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") "
Nov 03 13:45:20.742536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.742434    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-tmp-dir\") pod \"495bb85e-709f-405d-a92e-e681113bb0e1\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") "
Nov 03 13:45:20.742605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.742542    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-model-cache\") pod \"495bb85e-709f-405d-a92e-e681113bb0e1\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") "
Nov 03 13:45:20.742676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.742621    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/495bb85e-709f-405d-a92e-e681113bb0e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/495bb85e-709f-405d-a92e-e681113bb0e1-tls-certs\") pod \"495bb85e-709f-405d-a92e-e681113bb0e1\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") "
Nov 03 13:45:20.742676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.742625    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-home" pod "495bb85e-709f-405d-a92e-e681113bb0e1" (UID: "495bb85e-709f-405d-a92e-e681113bb0e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:20.742676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.742668    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495bb85e-709f-405d-a92e-e681113bb0e1-kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/495bb85e-709f-405d-a92e-e681113bb0e1-kube-api-access-8dqg5\") pod \"495bb85e-709f-405d-a92e-e681113bb0e1\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") "
Nov 03 13:45:20.742851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.742716    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-dshm\") pod \"495bb85e-709f-405d-a92e-e681113bb0e1\" (UID: \"495bb85e-709f-405d-a92e-e681113bb0e1\") "
Nov 03 13:45:20.742851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.742751    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-tmp-dir" pod "495bb85e-709f-405d-a92e-e681113bb0e1" (UID: "495bb85e-709f-405d-a92e-e681113bb0e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:20.742851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.742792    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-model-cache" pod "495bb85e-709f-405d-a92e-e681113bb0e1" (UID: "495bb85e-709f-405d-a92e-e681113bb0e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:20.743119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.743096    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.743259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.743125    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.743259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.743140    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.746251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.745040    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495bb85e-709f-405d-a92e-e681113bb0e1-kube-api-access-8dqg5" pod "495bb85e-709f-405d-a92e-e681113bb0e1" (UID: "495bb85e-709f-405d-a92e-e681113bb0e1"). InnerVolumeSpecName "kube-api-access-8dqg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:20.746251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.745206    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-dshm" pod "495bb85e-709f-405d-a92e-e681113bb0e1" (UID: "495bb85e-709f-405d-a92e-e681113bb0e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:20.746251 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.745295    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495bb85e-709f-405d-a92e-e681113bb0e1-tls-certs" pod "495bb85e-709f-405d-a92e-e681113bb0e1" (UID: "495bb85e-709f-405d-a92e-e681113bb0e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:45:20.843935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.843904    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-model-cache\") pod \"fbbb9af9-1713-44f4-afcd-901667677352\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") "
Nov 03 13:45:20.844119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.843944    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-tmp-dir\") pod \"fbbb9af9-1713-44f4-afcd-901667677352\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") "
Nov 03 13:45:20.844119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.843969    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-dshm\") pod \"fbbb9af9-1713-44f4-afcd-901667677352\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") "
Nov 03 13:45:20.844119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844011    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbb9af9-1713-44f4-afcd-901667677352-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbb9af9-1713-44f4-afcd-901667677352-tls-certs\") pod \"fbbb9af9-1713-44f4-afcd-901667677352\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") "
Nov 03 13:45:20.844354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844165    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-home\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-home\") pod \"fbbb9af9-1713-44f4-afcd-901667677352\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") "
Nov 03 13:45:20.844354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844225    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-tmp-dir" pod "fbbb9af9-1713-44f4-afcd-901667677352" (UID: "fbbb9af9-1713-44f4-afcd-901667677352"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:20.844354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844244    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-model-cache" pod "fbbb9af9-1713-44f4-afcd-901667677352" (UID: "fbbb9af9-1713-44f4-afcd-901667677352"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:20.844354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844233    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbb9af9-1713-44f4-afcd-901667677352-kube-api-access-xpq6x\" (UniqueName: \"kubernetes.io/projected/fbbb9af9-1713-44f4-afcd-901667677352-kube-api-access-xpq6x\") pod \"fbbb9af9-1713-44f4-afcd-901667677352\" (UID: \"fbbb9af9-1713-44f4-afcd-901667677352\") "
Nov 03 13:45:20.844555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844435    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-home" pod "fbbb9af9-1713-44f4-afcd-901667677352" (UID: "fbbb9af9-1713-44f4-afcd-901667677352"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:20.844706 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844687    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495bb85e-709f-405d-a92e-e681113bb0e1-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.844780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844714    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dqg5\" (UniqueName: \"kubernetes.io/projected/495bb85e-709f-405d-a92e-e681113bb0e1-kube-api-access-8dqg5\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.844780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844729    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495bb85e-709f-405d-a92e-e681113bb0e1-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.844780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844743    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.844780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844756    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.844780 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.844770    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.846076 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.846056    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-dshm" pod "fbbb9af9-1713-44f4-afcd-901667677352" (UID: "fbbb9af9-1713-44f4-afcd-901667677352"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:45:20.846151 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.846079    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbb9af9-1713-44f4-afcd-901667677352-tls-certs" pod "fbbb9af9-1713-44f4-afcd-901667677352" (UID: "fbbb9af9-1713-44f4-afcd-901667677352"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:45:20.846462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.846447    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbb9af9-1713-44f4-afcd-901667677352-kube-api-access-xpq6x" pod "fbbb9af9-1713-44f4-afcd-901667677352" (UID: "fbbb9af9-1713-44f4-afcd-901667677352"). InnerVolumeSpecName "kube-api-access-xpq6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:20.946347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.946275    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbb9af9-1713-44f4-afcd-901667677352-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.946347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.946302    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbb9af9-1713-44f4-afcd-901667677352-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:20.946347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:20.946317    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpq6x\" (UniqueName: \"kubernetes.io/projected/fbbb9af9-1713-44f4-afcd-901667677352-kube-api-access-xpq6x\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:21.084642 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.084603    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" path="/var/lib/kubelet/pods/17f84e6b-bd58-4d15-a856-2900abe93ec4/volumes"
Nov 03 13:45:21.085290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.085245    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" path="/var/lib/kubelet/pods/81842164-51e8-4b5b-b59d-1e5a50fc94ab/volumes"
Nov 03 13:45:21.128419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.128384    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:21.575274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.575245    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"
Nov 03 13:45:21.575274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.575255    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv" event={"ID":"fbbb9af9-1713-44f4-afcd-901667677352","Type":"ContainerDied","Data":"1f6cf6f58940f9ee745cc7a1220bd8ba5f5bac9645513aacacd70f45574a1d3d"}
Nov 03 13:45:21.575764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.575313    5420 scope.go:122] "RemoveContainer" containerID="8b93dc9d868d44062d10e5d2bb91229f923a55c36595551e5c516b1ea64e16ec"
Nov 03 13:45:21.579229 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.579198    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz" event={"ID":"495bb85e-709f-405d-a92e-e681113bb0e1","Type":"ContainerDied","Data":"d1ca8a3229a4ac9a8ea89060fa42c011d931fa7675fc002e8817d6b3ef5a6795"}
Nov 03 13:45:21.579229 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.579206    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"
Nov 03 13:45:21.585303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.585289    5420 scope.go:122] "RemoveContainer" containerID="8c697e6621871ca1721c3a4d494a5f7fc6bcec34fdc5b5e86ef53c9685a586d6"
Nov 03 13:45:21.598518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.598489    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"]
Nov 03 13:45:21.602541 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.602521    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/simulator-w0-a4ab8e-kserve-75c76d88c6-tl5vv"]
Nov 03 13:45:21.619129 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.619099    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"]
Nov 03 13:45:21.623045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.623025    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a4ab8e/premium-w0-a4ab8e-kserve-57f6b9c78f-j42kz"]
Nov 03 13:45:21.931281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:21.931229    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:22.091602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:22.091570    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:22.802307 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:22.802278    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:22.946914 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:22.946889    5420 scope.go:122] "RemoveContainer" containerID="2b0f5c493a71f346be91da05005169b7a71b0b9e2913ca2fe1725ab9748f3ed1"
Nov 03 13:45:22.956098 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:22.956077    5420 scope.go:122] "RemoveContainer" containerID="8eb7aa7ed7930fa79728ca6431034b3975d219f5f9b029f8ad91b77bf1e01048"
Nov 03 13:45:22.964335 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:22.964318    5420 scope.go:122] "RemoveContainer" containerID="d96a558da369378be0f9e68cb432b1d082269f4a02c6976b05970d7ecd398b17"
Nov 03 13:45:22.972102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:22.972082    5420 scope.go:122] "RemoveContainer" containerID="6bfbfe221bf31d869f9b569d82a8da2cd0b8f647844bc5b0ae202cb850fbe6f1"
Nov 03 13:45:23.002086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.002060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:23.082939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.082885    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" path="/var/lib/kubelet/pods/495bb85e-709f-405d-a92e-e681113bb0e1/volumes"
Nov 03 13:45:23.083315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.083301    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbb9af9-1713-44f4-afcd-901667677352" path="/var/lib/kubelet/pods/fbbb9af9-1713-44f4-afcd-901667677352/volumes"
Nov 03 13:45:23.091964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.091946    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"]
Nov 03 13:45:23.092398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092384    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerName="cleanup"
Nov 03 13:45:23.092451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092400    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerName="cleanup"
Nov 03 13:45:23.092451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092414    5420 cpu_manager.go:414] "containerMap: removing container" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerName="cleanup"
Nov 03 13:45:23.092451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092420    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerName="cleanup"
Nov 03 13:45:23.092451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092428    5420 cpu_manager.go:414] "containerMap: removing container" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerName="cleanup"
Nov 03 13:45:23.092451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092433    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerName="cleanup"
Nov 03 13:45:23.092451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092439    5420 cpu_manager.go:414] "containerMap: removing container" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy"
Nov 03 13:45:23.092451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092443    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092455    5420 cpu_manager.go:414] "containerMap: removing container" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerName="cleanup"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092459    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerName="cleanup"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092468    5420 cpu_manager.go:414] "containerMap: removing container" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerName="cleanup"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092473    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerName="cleanup"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092482    5420 cpu_manager.go:414] "containerMap: removing container" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerName="cleanup"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092486    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerName="cleanup"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092498    5420 cpu_manager.go:414] "containerMap: removing container" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092502    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092508    5420 cpu_manager.go:414] "containerMap: removing container" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092512    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092518    5420 cpu_manager.go:414] "containerMap: removing container" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerName="cleanup"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092523    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerName="cleanup"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092533    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerName="cleanup"
Nov 03 13:45:23.092695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092537    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerName="cleanup"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092703    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerName="cleanup"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092715    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbb9af9-1713-44f4-afcd-901667677352" containerName="main"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092720    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="495bb85e-709f-405d-a92e-e681113bb0e1" containerName="main"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092727    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerName="cleanup"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092733    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerName="cleanup"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092739    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerName="cleanup"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092746    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerName="cleanup"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092752    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerName="cleanup"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092757    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f84e6b-bd58-4d15-a856-2900abe93ec4" containerName="istio-proxy"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092763    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="81842164-51e8-4b5b-b59d-1e5a50fc94ab" containerName="cleanup"
Nov 03 13:45:23.093154 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.092768    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerName="cleanup"
Nov 03 13:45:23.097999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.097983    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.107872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.107850    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:45:23.108007 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.107984    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:45:23.109568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.109546    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:45:23.109568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.109551    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:45:23.109943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.109925    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"]
Nov 03 13:45:23.163998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.163967    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"]
Nov 03 13:45:23.169329 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.169297    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svp67\" (UniqueName: \"kubernetes.io/projected/e597cbfb-fc62-4004-b866-1949463f3507-kube-api-access-svp67\") pod \"maas-api-revoke-sub-f8e60b36-vhvxq\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.169509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.169357    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e60b36-vhvxq\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.169509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.169458    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e60b36-vhvxq\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.209587 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.209540    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:23.270623 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.270594    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e60b36-vhvxq\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.270793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.270661    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e60b36-vhvxq\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.270793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.270766    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svp67\" (UniqueName: \"kubernetes.io/projected/e597cbfb-fc62-4004-b866-1949463f3507-kube-api-access-svp67\") pod \"maas-api-revoke-sub-f8e60b36-vhvxq\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.271267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.271243    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8e60b36-vhvxq\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.272908 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.272884    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8e60b36-vhvxq\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.283638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.283613    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svp67\" (UniqueName: \"kubernetes.io/projected/e597cbfb-fc62-4004-b866-1949463f3507-kube-api-access-svp67\") pod \"maas-api-revoke-sub-f8e60b36-vhvxq\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.387278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.387171    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:23.412404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.412381    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:23.720221 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.720179    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"]
Nov 03 13:45:23.721613 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:45:23.721580    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode597cbfb_fc62_4004_b866_1949463f3507.slice/crio-71a24aed6c3914e7b29701dc7419ef9d46b424ae23c52eea920fe5477b5d36c1 WatchSource:0}: Error finding container 71a24aed6c3914e7b29701dc7419ef9d46b424ae23c52eea920fe5477b5d36c1: Status 404 returned error can't find the container with id 71a24aed6c3914e7b29701dc7419ef9d46b424ae23c52eea920fe5477b5d36c1
Nov 03 13:45:23.905574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:23.905544    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:24.601792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:24.601759    5420 generic.go:356] "Generic (PLEG): container finished" podID="e597cbfb-fc62-4004-b866-1949463f3507" containerID="d41bd9da2616cb32a2e3fcf79ab93343a8874016a05f0608787bd28d5dcfdb8c" exitCode=0
Nov 03 13:45:24.602001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:24.601842    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq" event={"ID":"e597cbfb-fc62-4004-b866-1949463f3507","Type":"ContainerDied","Data":"d41bd9da2616cb32a2e3fcf79ab93343a8874016a05f0608787bd28d5dcfdb8c"}
Nov 03 13:45:24.602001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:24.601895    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq" event={"ID":"e597cbfb-fc62-4004-b866-1949463f3507","Type":"ContainerStarted","Data":"71a24aed6c3914e7b29701dc7419ef9d46b424ae23c52eea920fe5477b5d36c1"}
Nov 03 13:45:24.730133 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:24.730098    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:45:24.887688 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:24.887656    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:25.269666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.269630    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:25.486274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.486243    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:25.723391 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.723364    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:25.743222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.743197    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:25.802079 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.802053    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-xks-ca\") pod \"e597cbfb-fc62-4004-b866-1949463f3507\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") "
Nov 03 13:45:25.802261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.802137    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e597cbfb-fc62-4004-b866-1949463f3507-kube-api-access-svp67\" (UniqueName: \"kubernetes.io/projected/e597cbfb-fc62-4004-b866-1949463f3507-kube-api-access-svp67\") pod \"e597cbfb-fc62-4004-b866-1949463f3507\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") "
Nov 03 13:45:25.802261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.802176    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-openshift-ca\") pod \"e597cbfb-fc62-4004-b866-1949463f3507\" (UID: \"e597cbfb-fc62-4004-b866-1949463f3507\") "
Nov 03 13:45:25.802626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.802588    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-openshift-ca" pod "e597cbfb-fc62-4004-b866-1949463f3507" (UID: "e597cbfb-fc62-4004-b866-1949463f3507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:25.804130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.804104    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-xks-ca" pod "e597cbfb-fc62-4004-b866-1949463f3507" (UID: "e597cbfb-fc62-4004-b866-1949463f3507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:45:25.804363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.804338    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e597cbfb-fc62-4004-b866-1949463f3507-kube-api-access-svp67" pod "e597cbfb-fc62-4004-b866-1949463f3507" (UID: "e597cbfb-fc62-4004-b866-1949463f3507"). InnerVolumeSpecName "kube-api-access-svp67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:25.903592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.903530    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:25.903592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.903553    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svp67\" (UniqueName: \"kubernetes.io/projected/e597cbfb-fc62-4004-b866-1949463f3507-kube-api-access-svp67\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:25.903592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:25.903564    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597cbfb-fc62-4004-b866-1949463f3507-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:26.543071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:26.542981    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:26.615043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:26.615015    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"
Nov 03 13:45:26.615207 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:26.615050    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq" event={"ID":"e597cbfb-fc62-4004-b866-1949463f3507","Type":"ContainerDied","Data":"71a24aed6c3914e7b29701dc7419ef9d46b424ae23c52eea920fe5477b5d36c1"}
Nov 03 13:45:26.615207 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:26.615099    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71a24aed6c3914e7b29701dc7419ef9d46b424ae23c52eea920fe5477b5d36c1"
Nov 03 13:45:27.264388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:27.264359    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:28.060591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:28.060558    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:28.164982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:28.164953    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p" podUID="a2aab2b3-fa9c-472e-b052-796540fc6e20" containerName="payload-processing" containerID="cri-o://4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f" gracePeriod=25
Nov 03 13:45:28.725027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:28.724996    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:29.491034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:29.491003    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:30.141094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:30.141063    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:30.440782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:30.440707    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"]
Nov 03 13:45:30.472900 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:30.472872    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"]
Nov 03 13:45:30.584971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:30.584945    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:30.949334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:30.949282    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:31.189035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:31.189001    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:31.365519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:31.365488    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:32.269868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:32.269838    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:33.108350 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:33.108312    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:45:33.119347 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:33.119319    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:33.295428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:33.295390    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:33.955573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:33.955541    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:34.680597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:34.680566    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:35.439418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:35.439389    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:35.473228 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:35.473176    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2" podUID="290a73e4-7a48-4e53-bb52-d49caa66f19f" containerName="payload-pre-processing" containerID="cri-o://8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d" gracePeriod=25
Nov 03 13:45:35.638561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:35.638524    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7" podUID="a1c61d8e-873c-4198-a6d3-b75706be6bf3" containerName="payload-processing" containerID="cri-o://12b8fcb0381da3c8fee87f3df375a227e884d001b413395138b2b22ed709e078" gracePeriod=25
Nov 03 13:45:36.151353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:36.151300    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:36.326494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:36.326458    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7" podUID="a1c61d8e-873c-4198-a6d3-b75706be6bf3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:36.878558 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:36.878526    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:37.688256 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:37.688204    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p" podUID="a2aab2b3-fa9c-472e-b052-796540fc6e20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:37.741673 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:37.741642    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:38.552802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:38.552768    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:39.117449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:39.117424    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:39.391637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:39.391568    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:45:39.439799 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:39.439767    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:40.206999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:40.206971    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:40.385714 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:40.385680    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:40.571656 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:40.571574    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:40.727451 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:40.727422    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:41.155365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:41.155336    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:41.382868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:41.382834    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:41.905740 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:41.905701    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:42.143751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:42.143722    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:42.357880 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:42.357850    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:43.116398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:43.116353    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:44.047181 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:44.047151    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:44.821528 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:44.821496    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:45.110326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:45.110244    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:45.305093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:45.305057    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2" podUID="290a73e4-7a48-4e53-bb52-d49caa66f19f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:45.642570 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:45.642537    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:46.295094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:46.295060    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:45:46.326568 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:46.326528    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7" podUID="a1c61d8e-873c-4198-a6d3-b75706be6bf3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:46.344869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:46.344838    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:47.186279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:47.186195    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:47.687816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:47.687779    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p" podUID="a2aab2b3-fa9c-472e-b052-796540fc6e20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:48.166281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.166247    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:48.407019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.406997    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:45:48.417028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.416972    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2aab2b3-fa9c-472e-b052-796540fc6e20-kube-api-access-hmbwd\" (UniqueName: \"kubernetes.io/projected/a2aab2b3-fa9c-472e-b052-796540fc6e20-kube-api-access-hmbwd\") pod \"a2aab2b3-fa9c-472e-b052-796540fc6e20\" (UID: \"a2aab2b3-fa9c-472e-b052-796540fc6e20\") "
Nov 03 13:45:48.417133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.417029    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2aab2b3-fa9c-472e-b052-796540fc6e20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2aab2b3-fa9c-472e-b052-796540fc6e20-plugins-config-volume\") pod \"a2aab2b3-fa9c-472e-b052-796540fc6e20\" (UID: \"a2aab2b3-fa9c-472e-b052-796540fc6e20\") "
Nov 03 13:45:48.417430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.417410    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2aab2b3-fa9c-472e-b052-796540fc6e20-plugins-config-volume" pod "a2aab2b3-fa9c-472e-b052-796540fc6e20" (UID: "a2aab2b3-fa9c-472e-b052-796540fc6e20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:48.418853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.418832    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2aab2b3-fa9c-472e-b052-796540fc6e20-kube-api-access-hmbwd" pod "a2aab2b3-fa9c-472e-b052-796540fc6e20" (UID: "a2aab2b3-fa9c-472e-b052-796540fc6e20"). InnerVolumeSpecName "kube-api-access-hmbwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:48.518733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.518704    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmbwd\" (UniqueName: \"kubernetes.io/projected/a2aab2b3-fa9c-472e-b052-796540fc6e20-kube-api-access-hmbwd\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:48.518733 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.518730    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2aab2b3-fa9c-472e-b052-796540fc6e20-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:48.743777 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.743748    5420 generic.go:356] "Generic (PLEG): container finished" podID="a2aab2b3-fa9c-472e-b052-796540fc6e20" containerID="4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f" exitCode=0
Nov 03 13:45:48.744259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.743787    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"
Nov 03 13:45:48.744259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.743824    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p" event={"ID":"a2aab2b3-fa9c-472e-b052-796540fc6e20","Type":"ContainerDied","Data":"4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f"}
Nov 03 13:45:48.744259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.743873    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p" event={"ID":"a2aab2b3-fa9c-472e-b052-796540fc6e20","Type":"ContainerDied","Data":"5c5a0d8220d11f4c2051a7b7526f3babefc4b695e495aac441c437460061bc5c"}
Nov 03 13:45:48.744259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.743894    5420 scope.go:122] "RemoveContainer" containerID="4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f"
Nov 03 13:45:48.753880 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.753864    5420 scope.go:122] "RemoveContainer" containerID="4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f"
Nov 03 13:45:48.754142 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:48.754125    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f\": container with ID starting with 4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f not found: ID does not exist" containerID="4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f"
Nov 03 13:45:48.754204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.754149    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f"} err="failed to get container status \"4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f\": rpc error: code = NotFound desc = could not find container \"4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f\": container with ID starting with 4960629bb0216141167301b3a617e6a2918c212bcfee40f8cf20774d670f115f not found: ID does not exist"
Nov 03 13:45:48.781750 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.781725    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"]
Nov 03 13:45:48.797101 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.797076    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a4ab8e-5fcd9fd57-cnx9p"]
Nov 03 13:45:48.995545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:48.995466    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"]
Nov 03 13:45:49.010084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:49.010050    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"]
Nov 03 13:45:49.027615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:49.027585    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:49.092511 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:49.092476    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2aab2b3-fa9c-472e-b052-796540fc6e20" path="/var/lib/kubelet/pods/a2aab2b3-fa9c-472e-b052-796540fc6e20/volumes"
Nov 03 13:45:49.217346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:49.217312    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:49.683751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:49.683722    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:49.851905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:49.851878    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:50.132681 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:50.132636    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:50.323406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:50.323373    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:50.744913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:50.744830    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:50.948137 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:50.948108    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:51.483309 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:51.483281    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:52.269481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:52.269453    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:52.430003 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:52.429970    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:52.768821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:52.768792    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:45:53.996400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:53.996337    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2" podUID="56237f7c-9a5e-4437-b33a-38d06e5cc1e2" containerName="payload-processing" containerID="cri-o://4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854" gracePeriod=25
Nov 03 13:45:54.010999 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:54.010970    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf" podUID="f83cde7e-d888-42b2-ba6b-73d80875a054" containerName="payload-pre-processing" containerID="cri-o://42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de" gracePeriod=25
Nov 03 13:45:54.730017 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:54.729979    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:45:55.305417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.305381    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2" podUID="290a73e4-7a48-4e53-bb52-d49caa66f19f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:55.765105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.765083    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:45:55.792885 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.792852    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290a73e4-7a48-4e53-bb52-d49caa66f19f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290a73e4-7a48-4e53-bb52-d49caa66f19f-plugins-config-volume\") pod \"290a73e4-7a48-4e53-bb52-d49caa66f19f\" (UID: \"290a73e4-7a48-4e53-bb52-d49caa66f19f\") "
Nov 03 13:45:55.793087 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.792969    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290a73e4-7a48-4e53-bb52-d49caa66f19f-kube-api-access-76lz4\" (UniqueName: \"kubernetes.io/projected/290a73e4-7a48-4e53-bb52-d49caa66f19f-kube-api-access-76lz4\") pod \"290a73e4-7a48-4e53-bb52-d49caa66f19f\" (UID: \"290a73e4-7a48-4e53-bb52-d49caa66f19f\") "
Nov 03 13:45:55.793503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.793474    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290a73e4-7a48-4e53-bb52-d49caa66f19f-plugins-config-volume" pod "290a73e4-7a48-4e53-bb52-d49caa66f19f" (UID: "290a73e4-7a48-4e53-bb52-d49caa66f19f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:55.795813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.795783    5420 generic.go:356] "Generic (PLEG): container finished" podID="a1c61d8e-873c-4198-a6d3-b75706be6bf3" containerID="12b8fcb0381da3c8fee87f3df375a227e884d001b413395138b2b22ed709e078" exitCode=0
Nov 03 13:45:55.795933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.795856    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7" event={"ID":"a1c61d8e-873c-4198-a6d3-b75706be6bf3","Type":"ContainerDied","Data":"12b8fcb0381da3c8fee87f3df375a227e884d001b413395138b2b22ed709e078"}
Nov 03 13:45:55.796446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.796421    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290a73e4-7a48-4e53-bb52-d49caa66f19f-kube-api-access-76lz4" pod "290a73e4-7a48-4e53-bb52-d49caa66f19f" (UID: "290a73e4-7a48-4e53-bb52-d49caa66f19f"). InnerVolumeSpecName "kube-api-access-76lz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:55.797409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.797383    5420 generic.go:356] "Generic (PLEG): container finished" podID="290a73e4-7a48-4e53-bb52-d49caa66f19f" containerID="8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d" exitCode=0
Nov 03 13:45:55.797514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.797422    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"
Nov 03 13:45:55.797514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.797457    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2" event={"ID":"290a73e4-7a48-4e53-bb52-d49caa66f19f","Type":"ContainerDied","Data":"8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d"}
Nov 03 13:45:55.797514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.797504    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2" event={"ID":"290a73e4-7a48-4e53-bb52-d49caa66f19f","Type":"ContainerDied","Data":"5a30e3babcb62d2389f42c476f732e87b3085cb7a1e6a1247978dbe695545daa"}
Nov 03 13:45:55.797682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.797533    5420 scope.go:122] "RemoveContainer" containerID="8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d"
Nov 03 13:45:55.808417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.808397    5420 scope.go:122] "RemoveContainer" containerID="8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d"
Nov 03 13:45:55.808700 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:45:55.808682    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d\": container with ID starting with 8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d not found: ID does not exist" containerID="8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d"
Nov 03 13:45:55.808769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.808709    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d"} err="failed to get container status \"8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d\": rpc error: code = NotFound desc = could not find container \"8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d\": container with ID starting with 8363952bbc61bd04dd2e4a0ba5f6669285b05f0d9f07c49824bafbc42e89e91d not found: ID does not exist"
Nov 03 13:45:55.822467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.822443    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"]
Nov 03 13:45:55.829756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.829732    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ded926-5d558d54f5-cwcn2"]
Nov 03 13:45:55.891898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.891878    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:45:55.893898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.893879    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290a73e4-7a48-4e53-bb52-d49caa66f19f-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:55.893979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.893900    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76lz4\" (UniqueName: \"kubernetes.io/projected/290a73e4-7a48-4e53-bb52-d49caa66f19f-kube-api-access-76lz4\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:55.994559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.994534    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c61d8e-873c-4198-a6d3-b75706be6bf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c61d8e-873c-4198-a6d3-b75706be6bf3-plugins-config-volume\") pod \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\" (UID: \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\") "
Nov 03 13:45:55.994711 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.994694    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c61d8e-873c-4198-a6d3-b75706be6bf3-kube-api-access-kj46x\" (UniqueName: \"kubernetes.io/projected/a1c61d8e-873c-4198-a6d3-b75706be6bf3-kube-api-access-kj46x\") pod \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\" (UID: \"a1c61d8e-873c-4198-a6d3-b75706be6bf3\") "
Nov 03 13:45:55.994952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.994932    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c61d8e-873c-4198-a6d3-b75706be6bf3-plugins-config-volume" pod "a1c61d8e-873c-4198-a6d3-b75706be6bf3" (UID: "a1c61d8e-873c-4198-a6d3-b75706be6bf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:45:55.995017 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.995003    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c61d8e-873c-4198-a6d3-b75706be6bf3-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:55.996726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:55.996699    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c61d8e-873c-4198-a6d3-b75706be6bf3-kube-api-access-kj46x" pod "a1c61d8e-873c-4198-a6d3-b75706be6bf3" (UID: "a1c61d8e-873c-4198-a6d3-b75706be6bf3"). InnerVolumeSpecName "kube-api-access-kj46x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:45:56.096008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:56.095983    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj46x\" (UniqueName: \"kubernetes.io/projected/a1c61d8e-873c-4198-a6d3-b75706be6bf3-kube-api-access-kj46x\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:45:56.804186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:56.804099    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"
Nov 03 13:45:56.804186 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:56.804119    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7" event={"ID":"a1c61d8e-873c-4198-a6d3-b75706be6bf3","Type":"ContainerDied","Data":"31f111aae97b3a750e7f8f08df7e9b7dfd91ce8708727147f7eb4e2d2f8011d5"}
Nov 03 13:45:56.804726 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:56.804190    5420 scope.go:122] "RemoveContainer" containerID="12b8fcb0381da3c8fee87f3df375a227e884d001b413395138b2b22ed709e078"
Nov 03 13:45:56.827563 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:56.827543    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"]
Nov 03 13:45:56.832757 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:56.832735    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ded926-79455f68bc-c64g7"]
Nov 03 13:45:57.082240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:57.082155    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290a73e4-7a48-4e53-bb52-d49caa66f19f" path="/var/lib/kubelet/pods/290a73e4-7a48-4e53-bb52-d49caa66f19f/volumes"
Nov 03 13:45:57.082574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:57.082558    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c61d8e-873c-4198-a6d3-b75706be6bf3" path="/var/lib/kubelet/pods/a1c61d8e-873c-4198-a6d3-b75706be6bf3/volumes"
Nov 03 13:45:57.629751 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:57.629716    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:57.769130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:57.769096    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:58.022815 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:58.022782    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"]
Nov 03 13:45:58.025204 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:58.025184    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee95c914-wxbq7"]
Nov 03 13:45:58.079984 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:58.079950    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:58.431665 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:58.431593    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:59.023870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:59.023842    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"]
Nov 03 13:45:59.027827 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:59.027810    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bafc04d1-w569l"]
Nov 03 13:45:59.053334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:59.053299    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf" podUID="f83cde7e-d888-42b2-ba6b-73d80875a054" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:59.053452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:59.053355    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2" podUID="56237f7c-9a5e-4437-b33a-38d06e5cc1e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:45:59.089960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:59.089932    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cebddea-d8a9-4939-a348-893d86da6489" path="/var/lib/kubelet/pods/1cebddea-d8a9-4939-a348-893d86da6489/volumes"
Nov 03 13:45:59.090351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:59.090338    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cfed13-1df9-4a89-985e-db71b0e6f87f" path="/var/lib/kubelet/pods/32cfed13-1df9-4a89-985e-db71b0e6f87f/volumes"
Nov 03 13:45:59.616648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:59.616617    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p5jpw_d8e7799f-b83f-48ff-b7c5-3baa8242f13e/manager/0.log"
Nov 03 13:45:59.816559 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:45:59.816522    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t5l28_6d127c5c-293c-4c38-8070-20b80d8c5efe/manager/0.log"
Nov 03 13:46:00.000395 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:00.000371    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n87db_bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef/kuadrant-console-plugin/0.log"
Nov 03 13:46:00.444687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:00.444602    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-fhvn2_d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63/limitador/0.log"
Nov 03 13:46:00.612074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:00.612046    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-tzjk2_440de568-380c-4201-95bc-d07af7d3d473/manager/0.log"
Nov 03 13:46:08.028720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:08.028682    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"]
Nov 03 13:46:08.031692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:08.031671    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487ba6f7-rp9jf"]
Nov 03 13:46:09.024472 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:09.024433    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"]
Nov 03 13:46:09.029823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:09.029795    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6857e6-sf28k"]
Nov 03 13:46:09.045050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:09.045021    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:46:09.052552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:09.052518    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf" podUID="f83cde7e-d888-42b2-ba6b-73d80875a054" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:46:09.052676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:09.052577    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2" podUID="56237f7c-9a5e-4437-b33a-38d06e5cc1e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:46:09.083977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:09.083948    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29f33dc-fd1f-40fc-8779-034d313df505" path="/var/lib/kubelet/pods/a29f33dc-fd1f-40fc-8779-034d313df505/volumes"
Nov 03 13:46:09.084419 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:09.084404    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d" path="/var/lib/kubelet/pods/d410728a-5f1d-4fd2-a5a7-c4d9e2e0222d/volumes"
Nov 03 13:46:11.049236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:11.048342    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"]
Nov 03 13:46:11.069232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:11.067154    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205bbc4e-tt9bb"]
Nov 03 13:46:11.099267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:11.099236    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94de26d1-fcbf-460d-aa5b-821b4647cb7a" path="/var/lib/kubelet/pods/94de26d1-fcbf-460d-aa5b-821b4647cb7a/volumes"
Nov 03 13:46:14.257410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.257337    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:46:14.288261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.288231    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:46:14.417812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.417781    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83cde7e-d888-42b2-ba6b-73d80875a054-kube-api-access-snq5x\" (UniqueName: \"kubernetes.io/projected/f83cde7e-d888-42b2-ba6b-73d80875a054-kube-api-access-snq5x\") pod \"f83cde7e-d888-42b2-ba6b-73d80875a054\" (UID: \"f83cde7e-d888-42b2-ba6b-73d80875a054\") "
Nov 03 13:46:14.417989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.417836    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-kube-api-access-89dvv\" (UniqueName: \"kubernetes.io/projected/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-kube-api-access-89dvv\") pod \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\" (UID: \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\") "
Nov 03 13:46:14.417989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.417917    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-plugins-config-volume\") pod \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\" (UID: \"56237f7c-9a5e-4437-b33a-38d06e5cc1e2\") "
Nov 03 13:46:14.417989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.417977    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83cde7e-d888-42b2-ba6b-73d80875a054-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cde7e-d888-42b2-ba6b-73d80875a054-plugins-config-volume\") pod \"f83cde7e-d888-42b2-ba6b-73d80875a054\" (UID: \"f83cde7e-d888-42b2-ba6b-73d80875a054\") "
Nov 03 13:46:14.418344 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.418319    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-plugins-config-volume" pod "56237f7c-9a5e-4437-b33a-38d06e5cc1e2" (UID: "56237f7c-9a5e-4437-b33a-38d06e5cc1e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:46:14.418446 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.418388    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83cde7e-d888-42b2-ba6b-73d80875a054-plugins-config-volume" pod "f83cde7e-d888-42b2-ba6b-73d80875a054" (UID: "f83cde7e-d888-42b2-ba6b-73d80875a054"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:46:14.419868 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.419848    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83cde7e-d888-42b2-ba6b-73d80875a054-kube-api-access-snq5x" pod "f83cde7e-d888-42b2-ba6b-73d80875a054" (UID: "f83cde7e-d888-42b2-ba6b-73d80875a054"). InnerVolumeSpecName "kube-api-access-snq5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:46:14.419933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.419855    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-kube-api-access-89dvv" pod "56237f7c-9a5e-4437-b33a-38d06e5cc1e2" (UID: "56237f7c-9a5e-4437-b33a-38d06e5cc1e2"). InnerVolumeSpecName "kube-api-access-89dvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:46:14.519659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.519584    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snq5x\" (UniqueName: \"kubernetes.io/projected/f83cde7e-d888-42b2-ba6b-73d80875a054-kube-api-access-snq5x\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:14.519659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.519613    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dvv\" (UniqueName: \"kubernetes.io/projected/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-kube-api-access-89dvv\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:14.519659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.519624    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56237f7c-9a5e-4437-b33a-38d06e5cc1e2-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:14.519659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.519633    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cde7e-d888-42b2-ba6b-73d80875a054-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:46:14.922882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.922792    5420 generic.go:356] "Generic (PLEG): container finished" podID="f83cde7e-d888-42b2-ba6b-73d80875a054" containerID="42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de" exitCode=0
Nov 03 13:46:14.922882 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.922845    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"
Nov 03 13:46:14.923128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.922876    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf" event={"ID":"f83cde7e-d888-42b2-ba6b-73d80875a054","Type":"ContainerDied","Data":"42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de"}
Nov 03 13:46:14.923128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.922935    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf" event={"ID":"f83cde7e-d888-42b2-ba6b-73d80875a054","Type":"ContainerDied","Data":"81565f3d0b1111522e812a5e4e8ea6bb2d73c734671c45161d13b88da78ba86b"}
Nov 03 13:46:14.923128 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.922960    5420 scope.go:122] "RemoveContainer" containerID="42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de"
Nov 03 13:46:14.924314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.924295    5420 generic.go:356] "Generic (PLEG): container finished" podID="56237f7c-9a5e-4437-b33a-38d06e5cc1e2" containerID="4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854" exitCode=0
Nov 03 13:46:14.924418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.924326    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2" event={"ID":"56237f7c-9a5e-4437-b33a-38d06e5cc1e2","Type":"ContainerDied","Data":"4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854"}
Nov 03 13:46:14.924418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.924345    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"
Nov 03 13:46:14.924418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.924387    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2" event={"ID":"56237f7c-9a5e-4437-b33a-38d06e5cc1e2","Type":"ContainerDied","Data":"94f63a87bd3515d88b5c75c2036c908c757c4e6535b07bc337f51551c5c32491"}
Nov 03 13:46:14.932770 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.932754    5420 scope.go:122] "RemoveContainer" containerID="42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de"
Nov 03 13:46:14.933077 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:46:14.933058    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de\": container with ID starting with 42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de not found: ID does not exist" containerID="42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de"
Nov 03 13:46:14.933134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.933084    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de"} err="failed to get container status \"42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de\": rpc error: code = NotFound desc = could not find container \"42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de\": container with ID starting with 42964692a846772a4de13425f3986af7f2d086c2ac3c3949fdfa498b9bb559de not found: ID does not exist"
Nov 03 13:46:14.933134 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.933101    5420 scope.go:122] "RemoveContainer" containerID="4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854"
Nov 03 13:46:14.941233 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.941191    5420 scope.go:122] "RemoveContainer" containerID="4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854"
Nov 03 13:46:14.941482 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:46:14.941465    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854\": container with ID starting with 4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854 not found: ID does not exist" containerID="4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854"
Nov 03 13:46:14.941552 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.941490    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854"} err="failed to get container status \"4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854\": rpc error: code = NotFound desc = could not find container \"4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854\": container with ID starting with 4b927f8293b1157476749bca55131a4c488934dc84e36f03d5c8f57e7dafe854 not found: ID does not exist"
Nov 03 13:46:14.953261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.953237    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"]
Nov 03 13:46:14.955555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.955534    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c192b-f6b6648b9-hrvh2"]
Nov 03 13:46:14.965071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.965049    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"]
Nov 03 13:46:14.969236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:14.969201    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c192b-7c64d44447-mdcqf"]
Nov 03 13:46:15.083140 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:15.083110    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56237f7c-9a5e-4437-b33a-38d06e5cc1e2" path="/var/lib/kubelet/pods/56237f7c-9a5e-4437-b33a-38d06e5cc1e2/volumes"
Nov 03 13:46:15.083521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:15.083507    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83cde7e-d888-42b2-ba6b-73d80875a054" path="/var/lib/kubelet/pods/f83cde7e-d888-42b2-ba6b-73d80875a054/volumes"
Nov 03 13:46:19.021192 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:19.021158    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"]
Nov 03 13:46:19.023412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:19.023393    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bb6759-4572h"]
Nov 03 13:46:19.082537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:19.082504    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90ea2f2-e22f-4ff6-9212-0a1c69998203" path="/var/lib/kubelet/pods/d90ea2f2-e22f-4ff6-9212-0a1c69998203/volumes"
Nov 03 13:46:19.887185 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:19.887155    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:46:20.154107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:20.154026    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:46:23.033375 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:23.033340    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"]
Nov 03 13:46:23.042387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:23.042362    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bab5a758-kgjdv"]
Nov 03 13:46:23.058406 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:23.058381    5420 scope.go:122] "RemoveContainer" containerID="d29e279166ec2b2f66fb2492c6c9258691dc36fc39cacd53cec309eb576b121e"
Nov 03 13:46:23.086265 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:23.086232    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e75044-127e-4acf-98be-f928956d6396" path="/var/lib/kubelet/pods/06e75044-127e-4acf-98be-f928956d6396/volumes"
Nov 03 13:46:24.730394 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:46:24.730359    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:46:30.027901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:30.027866    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"]
Nov 03 13:46:30.030763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:30.030739    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca8c2fb-bhqt4"]
Nov 03 13:46:31.083096 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:31.083062    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a16df6-6862-43c5-b438-9a6741980520" path="/var/lib/kubelet/pods/08a16df6-6862-43c5-b438-9a6741980520/volumes"
Nov 03 13:46:33.351274 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:33.351237    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:46:40.272021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:40.271981    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:46:40.426968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:40.426939    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:46:42.619093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:42.619053    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:46:44.484390 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:44.484359    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:46:54.730247 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:46:54.730186    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:46:57.790980 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:46:57.790896    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:47:15.680024 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:15.679980    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:47:15.826527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:15.826489    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:47:19.922872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:19.922844    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:47:20.185369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:20.185276    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:47:23.103281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:23.103248    5420 scope.go:122] "RemoveContainer" containerID="ea7bb5a9c3693e3c8cc12264b96572581f5051ea76154878cb3952e52e3458ec"
Nov 03 13:47:23.112484 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:23.112464    5420 scope.go:122] "RemoveContainer" containerID="1e7cf891ba826827001425ef9d6a92d428b36ee87073217dc618d68d9927660f"
Nov 03 13:47:23.121070 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:23.121048    5420 scope.go:122] "RemoveContainer" containerID="34df9cc96f22e11fdabe90775c7214f17ef1d7bfa565b34f62a1c71fda4917b2"
Nov 03 13:47:23.129260 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:23.129239    5420 scope.go:122] "RemoveContainer" containerID="8db4fab932c181e5e9af152957b373f1212c8acfda52fceaddb0129e90bb51e4"
Nov 03 13:47:23.137536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:23.137513    5420 scope.go:122] "RemoveContainer" containerID="d90f8f9b2d0c5ae2b19c913bb6195308c227cf13ab5aac981c990c63dbdd1ab4"
Nov 03 13:47:23.145979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:23.145956    5420 scope.go:122] "RemoveContainer" containerID="d707a4587f7a220d183386ee44213e007dd8cf220935143acf1f43d1adca483c"
Nov 03 13:47:23.157065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:47:23.157040    5420 scope.go:122] "RemoveContainer" containerID="cae3154cf953fd324cdb4edf2e0f116247469173f1c0ccaecf00e68dedf7390d"
Nov 03 13:47:24.731180 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:47:24.731143    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:47:54.730493 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:47:54.730454    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:48:17.997506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:17.997469    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:48:19.985036 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:19.985001    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:48:20.252197 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:20.252114    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:48:23.200410 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:23.200380    5420 scope.go:122] "RemoveContainer" containerID="a1b92f2818f16027353549364caa202fac6dd3158d6e46699f7bdd5dc62fd35b"
Nov 03 13:48:24.730004 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:48:24.729973    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:48:28.023388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:28.023315    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"]
Nov 03 13:48:28.027578 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:28.027552    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73fd1243-872p9"]
Nov 03 13:48:29.022785 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:29.022745    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"]
Nov 03 13:48:29.027832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:29.027803    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3829d1a7-568dd"]
Nov 03 13:48:29.083699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:29.083665    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdafece-fcbf-473a-b5e3-818b4761abcb" path="/var/lib/kubelet/pods/afdafece-fcbf-473a-b5e3-818b4761abcb/volumes"
Nov 03 13:48:29.084092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:29.084078    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baba52f8-142f-498f-a2c4-ed267f98a98f" path="/var/lib/kubelet/pods/baba52f8-142f-498f-a2c4-ed267f98a98f/volumes"
Nov 03 13:48:35.251342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:35.251294    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:48:39.022813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:39.022781    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"]
Nov 03 13:48:39.027379 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:39.027353    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2bff93-mhttp"]
Nov 03 13:48:39.082324 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:39.082299    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e57bec-211a-4e21-913e-f85c806a45bf" path="/var/lib/kubelet/pods/62e57bec-211a-4e21-913e-f85c806a45bf/volumes"
Nov 03 13:48:42.020945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:42.020910    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"]
Nov 03 13:48:42.023573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:42.023551    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3547802b-szkgb"]
Nov 03 13:48:43.082267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:43.082233    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a" path="/var/lib/kubelet/pods/0db6e2e6-07ac-4d9c-9d0b-fb7746afd36a/volumes"
Nov 03 13:48:50.469118 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:50.469082    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:48:54.718372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:54.718339    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:48:54.804606 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:48:54.804571    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:48:58.740538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:48:58.740498    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:49:00.059120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:00.059088    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:49:01.129080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:01.129044    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:49:02.021669 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:02.021633    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"]
Nov 03 13:49:02.023592 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:02.023568    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3b8a3fc-t28d2"]
Nov 03 13:49:03.084909 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:03.084871    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00298a04-971e-4bc0-a25f-b0bd162681f9" path="/var/lib/kubelet/pods/00298a04-971e-4bc0-a25f-b0bd162681f9/volumes"
Nov 03 13:49:08.879166 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:08.879061    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:49:14.022369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:14.022338    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"]
Nov 03 13:49:14.024190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:14.024167    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a09b7f6-4frs5"]
Nov 03 13:49:15.090492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:15.090455    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666592f1-c7c3-4d29-8ac3-2e8611d94826" path="/var/lib/kubelet/pods/666592f1-c7c3-4d29-8ac3-2e8611d94826/volumes"
Nov 03 13:49:19.648767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:19.648735    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:49:19.659493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:19.659475    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:49:20.038321 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:20.038274    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:49:20.165633 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:20.165603    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:49:24.249458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:24.249430    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:49:24.737235 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:49:24.737160    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:49:35.027876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:35.027842    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"]
Nov 03 13:49:35.031316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:35.031291    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a24258-98rwv"]
Nov 03 13:49:35.082869 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:35.082838    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb73c91-ae95-4d53-91bf-27ae02c854ce" path="/var/lib/kubelet/pods/8eb73c91-ae95-4d53-91bf-27ae02c854ce/volumes"
Nov 03 13:49:37.946467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:37.946433    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:49:45.024045 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:45.024003    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"]
Nov 03 13:49:45.026801 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:45.026777    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ca9ee2-xpt5s"]
Nov 03 13:49:45.082086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:49:45.082053    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb" path="/var/lib/kubelet/pods/eb7bf45b-a4e4-4a1a-b6f5-fafec399b6fb/volumes"
Nov 03 13:49:54.730122 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:49:54.730086    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:50:05.426864 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.426775    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"]
Nov 03 13:50:05.427687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.427651    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs" podUID="7cf961da-3329-4396-96f5-817279fd5f03" containerName="discovery" containerID="cri-o://23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d" gracePeriod=30
Nov 03 13:50:05.714756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.714724    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:50:05.771398 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.771360    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-dns-cert\") pod \"7cf961da-3329-4396-96f5-817279fd5f03\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") "
Nov 03 13:50:05.771596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.771443    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-kubeconfig\") pod \"7cf961da-3329-4396-96f5-817279fd5f03\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") "
Nov 03 13:50:05.771596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.771475    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-ca-configmap\") pod \"7cf961da-3329-4396-96f5-817279fd5f03\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") "
Nov 03 13:50:05.771596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.771532    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-istio-token\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-istio-token\") pod \"7cf961da-3329-4396-96f5-817279fd5f03\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") "
Nov 03 13:50:05.771596 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.771573    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-cacerts\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-cacerts\") pod \"7cf961da-3329-4396-96f5-817279fd5f03\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") "
Nov 03 13:50:05.771847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.771610    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-kube-api-access-d62sl\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-kube-api-access-d62sl\") pod \"7cf961da-3329-4396-96f5-817279fd5f03\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") "
Nov 03 13:50:05.771847 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.771699    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf961da-3329-4396-96f5-817279fd5f03-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf961da-3329-4396-96f5-817279fd5f03-local-certs\") pod \"7cf961da-3329-4396-96f5-817279fd5f03\" (UID: \"7cf961da-3329-4396-96f5-817279fd5f03\") "
Nov 03 13:50:05.772952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.772680    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-ca-configmap" pod "7cf961da-3329-4396-96f5-817279fd5f03" (UID: "7cf961da-3329-4396-96f5-817279fd5f03"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:50:05.774638 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.774607    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-dns-cert" pod "7cf961da-3329-4396-96f5-817279fd5f03" (UID: "7cf961da-3329-4396-96f5-817279fd5f03"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:50:05.775019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.774991    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-kubeconfig" pod "7cf961da-3329-4396-96f5-817279fd5f03" (UID: "7cf961da-3329-4396-96f5-817279fd5f03"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:50:05.775522 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.775493    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-kube-api-access-d62sl" pod "7cf961da-3329-4396-96f5-817279fd5f03" (UID: "7cf961da-3329-4396-96f5-817279fd5f03"). InnerVolumeSpecName "kube-api-access-d62sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:50:05.775816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.775790    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-istio-token" pod "7cf961da-3329-4396-96f5-817279fd5f03" (UID: "7cf961da-3329-4396-96f5-817279fd5f03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:50:05.777086 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.777049    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-cacerts" pod "7cf961da-3329-4396-96f5-817279fd5f03" (UID: "7cf961da-3329-4396-96f5-817279fd5f03"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:50:05.777602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.777578    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf961da-3329-4396-96f5-817279fd5f03-local-certs" pod "7cf961da-3329-4396-96f5-817279fd5f03" (UID: "7cf961da-3329-4396-96f5-817279fd5f03"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:50:05.873294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.873265    5420 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf961da-3329-4396-96f5-817279fd5f03-local-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:50:05.873294 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.873292    5420 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-dns-cert\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:50:05.873594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.873305    5420 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-istio-kubeconfig\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:50:05.873594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.873317    5420 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cf961da-3329-4396-96f5-817279fd5f03-istio-csr-ca-configmap\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:50:05.873594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.873331    5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-istio-token\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:50:05.873594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.873343    5420 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cf961da-3329-4396-96f5-817279fd5f03-cacerts\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:50:05.873594 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:05.873354    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d62sl\" (UniqueName: \"kubernetes.io/projected/7cf961da-3329-4396-96f5-817279fd5f03-kube-api-access-d62sl\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:50:06.279235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:06.278271    5420 generic.go:356] "Generic (PLEG): container finished" podID="7cf961da-3329-4396-96f5-817279fd5f03" containerID="23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d" exitCode=0
Nov 03 13:50:06.279235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:06.278345    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs" event={"ID":"7cf961da-3329-4396-96f5-817279fd5f03","Type":"ContainerDied","Data":"23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d"}
Nov 03 13:50:06.279235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:06.278393    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs" event={"ID":"7cf961da-3329-4396-96f5-817279fd5f03","Type":"ContainerDied","Data":"5a76b968e6211910c1da441d222d13f6258eae4e799afba85fca89a603bfcad1"}
Nov 03 13:50:06.279235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:06.278420    5420 scope.go:122] "RemoveContainer" containerID="23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d"
Nov 03 13:50:06.279235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:06.278586    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"
Nov 03 13:50:06.309367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:06.309301    5420 scope.go:122] "RemoveContainer" containerID="23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d"
Nov 03 13:50:06.309621 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:50:06.309601    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d\": container with ID starting with 23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d not found: ID does not exist" containerID="23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d"
Nov 03 13:50:06.309664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:06.309633    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d"} err="failed to get container status \"23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d\": rpc error: code = NotFound desc = could not find container \"23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d\": container with ID starting with 23cca33f40aeb53c032d94702630d48b33a3dad191da0742b4bda7785d14eb4d not found: ID does not exist"
Nov 03 13:50:06.376236 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:06.355179    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"]
Nov 03 13:50:06.376508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:06.376482    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dvwqs"]
Nov 03 13:50:07.093188 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:07.093144    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf961da-3329-4396-96f5-817279fd5f03" path="/var/lib/kubelet/pods/7cf961da-3329-4396-96f5-817279fd5f03/volumes"
Nov 03 13:50:17.341315 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:17.341283    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:50:20.077412 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:20.077383    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:50:20.214894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:20.214860    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:50:23.230661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:23.230629    5420 scope.go:122] "RemoveContainer" containerID="f2bdb7ff34751b538fd609e777935f152decef12bc667bd296a44f4bb7b10069"
Nov 03 13:50:23.239088 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:23.239065    5420 scope.go:122] "RemoveContainer" containerID="14515f7b06d9939b003abb0593e746c95d2f97ff175d4b9e1d07a6feec8c4993"
Nov 03 13:50:23.247139 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:23.247123    5420 scope.go:122] "RemoveContainer" containerID="b85280e67600a7d01b61cdb738b8e7842d287e553d5e451467bc58361f1590ac"
Nov 03 13:50:23.254626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:23.254607    5420 scope.go:122] "RemoveContainer" containerID="347a8876ee51906af9f345d3b4840edaad6033fa940f61490be21a5f8acfd72c"
Nov 03 13:50:23.262193 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:23.262175    5420 scope.go:122] "RemoveContainer" containerID="21f021823c679c6b6ae81d7856daf1d80aaf2b483eeb429945cb91ed7d236cbb"
Nov 03 13:50:23.269148 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:23.269133    5420 scope.go:122] "RemoveContainer" containerID="9f1005d6e6ec0fa02b7a90dd452cc74f1fe7f968bf1f2681cd0d8a083fed9500"
Nov 03 13:50:24.730034 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:50:24.730003    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:50:26.034021 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:26.033982    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"]
Nov 03 13:50:26.037828 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:26.037803    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8e60b36-vhvxq"]
Nov 03 13:50:27.094664 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:27.094619    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e597cbfb-fc62-4004-b866-1949463f3507" path="/var/lib/kubelet/pods/e597cbfb-fc62-4004-b866-1949463f3507/volumes"
Nov 03 13:50:54.730054 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:50:54.730013    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:50:58.565074 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:58.565002    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:50:58.735050 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:50:58.735011    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:16.253855 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:16.253812    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:51:20.141172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:20.141134    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:21.797414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:21.797380    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:22.216444 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:22.216410    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:22.386308 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:22.386277    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:23.307588 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:23.307559    5420 scope.go:122] "RemoveContainer" containerID="93da930c9b91fe163668fe1be30ff5f91d6a9e15bd4e808608c95eaf1b5d9986"
Nov 03 13:51:23.317516 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:23.317490    5420 scope.go:122] "RemoveContainer" containerID="65f211f08b8166b8c1f25c2847da0161b33c7b0e3d9abd6d9bd0202ad24854f4"
Nov 03 13:51:24.731081 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:51:24.731046    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:51:24.823859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:24.823811    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:24.966720 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:24.966688    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:26.068686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:26.068649    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:26.226632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:26.226601    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:26.965659 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:26.965629    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:27.939152 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:27.939122    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:28.641345 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:28.641315    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:51:28.691316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:28.691283    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:29.579736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:29.579709    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:30.476977 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:30.476936    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:31.388071 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:31.388021    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:32.183846 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:32.183815    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:32.976822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:32.976786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:33.811965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:33.811933    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:34.671440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:34.671402    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:35.436013 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.435972    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:35.515330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515293    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"]
Nov 03 13:51:35.515745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515728    5420 cpu_manager.go:414] "containerMap: removing container" podUID="7cf961da-3329-4396-96f5-817279fd5f03" containerName="discovery"
Nov 03 13:51:35.515861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515747    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf961da-3329-4396-96f5-817279fd5f03" containerName="discovery"
Nov 03 13:51:35.515861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515763    5420 cpu_manager.go:414] "containerMap: removing container" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerName="cleanup"
Nov 03 13:51:35.515861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515771    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cdaa0a-1931-4f1e-bb21-aaa659790c1b" containerName="cleanup"
Nov 03 13:51:35.515861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515789    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e597cbfb-fc62-4004-b866-1949463f3507" containerName="revoke-keys"
Nov 03 13:51:35.515861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515796    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e597cbfb-fc62-4004-b866-1949463f3507" containerName="revoke-keys"
Nov 03 13:51:35.515861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515811    5420 cpu_manager.go:414] "containerMap: removing container" podUID="a1c61d8e-873c-4198-a6d3-b75706be6bf3" containerName="payload-processing"
Nov 03 13:51:35.515861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515819    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c61d8e-873c-4198-a6d3-b75706be6bf3" containerName="payload-processing"
Nov 03 13:51:35.515861 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515858    5420 cpu_manager.go:414] "containerMap: removing container" podUID="290a73e4-7a48-4e53-bb52-d49caa66f19f" containerName="payload-pre-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515865    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290a73e4-7a48-4e53-bb52-d49caa66f19f" containerName="payload-pre-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515881    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f83cde7e-d888-42b2-ba6b-73d80875a054" containerName="payload-pre-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515889    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83cde7e-d888-42b2-ba6b-73d80875a054" containerName="payload-pre-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515899    5420 cpu_manager.go:414] "containerMap: removing container" podUID="56237f7c-9a5e-4437-b33a-38d06e5cc1e2" containerName="payload-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515905    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56237f7c-9a5e-4437-b33a-38d06e5cc1e2" containerName="payload-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515918    5420 cpu_manager.go:414] "containerMap: removing container" podUID="a2aab2b3-fa9c-472e-b052-796540fc6e20" containerName="payload-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.515925    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2aab2b3-fa9c-472e-b052-796540fc6e20" containerName="payload-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.516025    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e597cbfb-fc62-4004-b866-1949463f3507" containerName="revoke-keys"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.516036    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c61d8e-873c-4198-a6d3-b75706be6bf3" containerName="payload-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.516052    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="690c206f-16a1-4f30-a7fb-0c2f9dd15ef3" containerName="cleanup"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.516061    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="290a73e4-7a48-4e53-bb52-d49caa66f19f" containerName="payload-pre-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.516070    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83cde7e-d888-42b2-ba6b-73d80875a054" containerName="payload-pre-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.516079    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="56237f7c-9a5e-4437-b33a-38d06e5cc1e2" containerName="payload-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.516088    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2aab2b3-fa9c-472e-b052-796540fc6e20" containerName="payload-processing"
Nov 03 13:51:35.516322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.516097    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf961da-3329-4396-96f5-817279fd5f03" containerName="discovery"
Nov 03 13:51:35.521130 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.521109    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:35.525783 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.525759    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:51:35.527481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.527458    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:51:35.527640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.527613    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\""
Nov 03 13:51:35.527640 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.527456    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:51:35.529595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.529570    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"]
Nov 03 13:51:35.567538 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.567504    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qxg\" (UniqueName: \"kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:35.567689 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.567548    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:35.567734 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.567688    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:35.668603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.668574    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qxg\" (UniqueName: \"kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:35.668774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.668618    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:35.668774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.668715    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:35.669388 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.669362    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:35.671110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:35.671088    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:35.675050 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:51:35.675025    5420 projected.go:196] Error preparing data for projected volume kube-api-access-74qxg for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 13:51:35.675392 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:51:35.675091    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg podName:306ba9c0-e04d-42e0-92d8-59a0650b0129 nodeName:}" failed. No retries permitted until 2026-11-03 13:51:36.175074991 +0000 UTC m=+2537.252839997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-74qxg" (UniqueName: "kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg") pod "maas-api-revoke-keys-e2e-ait-d969013e-thtzq" (UID: "306ba9c0-e04d-42e0-92d8-59a0650b0129") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 13:51:36.275179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:36.275137    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qxg\" (UniqueName: \"kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:36.280017 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:51:36.279996    5420 projected.go:196] Error preparing data for projected volume kube-api-access-74qxg for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 13:51:36.280101 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:51:36.280069    5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg podName:306ba9c0-e04d-42e0-92d8-59a0650b0129 nodeName:}" failed. No retries permitted until 2026-11-03 13:51:37.280055627 +0000 UTC m=+2538.357820635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-74qxg" (UniqueName: "kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg") pod "maas-api-revoke-keys-e2e-ait-d969013e-thtzq" (UID: "306ba9c0-e04d-42e0-92d8-59a0650b0129") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 13:51:36.543555 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:36.543475    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:37.263641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.263600    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:37.285025 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.284993    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qxg\" (UniqueName: \"kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:37.294285 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.294255    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qxg\" (UniqueName: \"kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg\") pod \"maas-api-revoke-keys-e2e-ait-d969013e-thtzq\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:37.336537 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.336506    5420 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq" secret="" err="secret \"maas-api-cleanup-dockercfg-phqfq\" not found"
Nov 03 13:51:37.336694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.336556    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:37.615232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.615190    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"]
Nov 03 13:51:37.617405 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:51:37.617375    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306ba9c0_e04d_42e0_92d8_59a0650b0129.slice/crio-30289ee9c1fe1a2ee750e5772f18b0af9ee331b43873926a108f4a5a07b495e1 WatchSource:0}: Error finding container 30289ee9c1fe1a2ee750e5772f18b0af9ee331b43873926a108f4a5a07b495e1: Status 404 returned error can't find the container with id 30289ee9c1fe1a2ee750e5772f18b0af9ee331b43873926a108f4a5a07b495e1
Nov 03 13:51:37.890939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.890864    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq" event={"ID":"306ba9c0-e04d-42e0-92d8-59a0650b0129","Type":"ContainerStarted","Data":"09edd8e4fe296643eda16b356057ff5dede99b58be8dc06795021e3551b41dd1"}
Nov 03 13:51:37.890939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.890917    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq" event={"ID":"306ba9c0-e04d-42e0-92d8-59a0650b0129","Type":"ContainerStarted","Data":"30289ee9c1fe1a2ee750e5772f18b0af9ee331b43873926a108f4a5a07b495e1"}
Nov 03 13:51:37.891196 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.891175    5420 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq" secret="" err="secret \"maas-api-cleanup-dockercfg-phqfq\" not found"
Nov 03 13:51:37.918127 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.913938    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq" podStartSLOduration=2.913919742 podStartE2EDuration="2.913919742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:37.910391567 +0000 UTC m=+2538.988156593" watchObservedRunningTime="2026-11-03 13:51:37.913919742 +0000 UTC m=+2538.991684768"
Nov 03 13:51:37.950998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:37.950917    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:38.764120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:38.764080    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:38.897573 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:38.897538    5420 generic.go:356] "Generic (PLEG): container finished" podID="306ba9c0-e04d-42e0-92d8-59a0650b0129" containerID="09edd8e4fe296643eda16b356057ff5dede99b58be8dc06795021e3551b41dd1" exitCode=0
Nov 03 13:51:38.897747 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:38.897609    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq" event={"ID":"306ba9c0-e04d-42e0-92d8-59a0650b0129","Type":"ContainerDied","Data":"09edd8e4fe296643eda16b356057ff5dede99b58be8dc06795021e3551b41dd1"}
Nov 03 13:51:39.602190 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:39.602155    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:40.040692 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.040669    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:40.112094 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.112065    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-openshift-ca\") pod \"306ba9c0-e04d-42e0-92d8-59a0650b0129\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") "
Nov 03 13:51:40.112331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.112128    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg\" (UniqueName: \"kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg\") pod \"306ba9c0-e04d-42e0-92d8-59a0650b0129\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") "
Nov 03 13:51:40.112331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.112295    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-xks-ca\") pod \"306ba9c0-e04d-42e0-92d8-59a0650b0129\" (UID: \"306ba9c0-e04d-42e0-92d8-59a0650b0129\") "
Nov 03 13:51:40.112467 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.112446    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-openshift-ca" pod "306ba9c0-e04d-42e0-92d8-59a0650b0129" (UID: "306ba9c0-e04d-42e0-92d8-59a0650b0129"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:51:40.112637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.112607    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:51:40.114298 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.114275    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg" pod "306ba9c0-e04d-42e0-92d8-59a0650b0129" (UID: "306ba9c0-e04d-42e0-92d8-59a0650b0129"). InnerVolumeSpecName "kube-api-access-74qxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:51:40.114387 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.114355    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-xks-ca" pod "306ba9c0-e04d-42e0-92d8-59a0650b0129" (UID: "306ba9c0-e04d-42e0-92d8-59a0650b0129"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:51:40.213955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.213924    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74qxg\" (UniqueName: \"kubernetes.io/projected/306ba9c0-e04d-42e0-92d8-59a0650b0129-kube-api-access-74qxg\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:51:40.213955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.213955    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ba9c0-e04d-42e0-92d8-59a0650b0129-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:51:40.376184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.376147    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:40.907562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.907536    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"
Nov 03 13:51:40.907725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.907534    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq" event={"ID":"306ba9c0-e04d-42e0-92d8-59a0650b0129","Type":"ContainerDied","Data":"30289ee9c1fe1a2ee750e5772f18b0af9ee331b43873926a108f4a5a07b495e1"}
Nov 03 13:51:40.907725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:40.907663    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30289ee9c1fe1a2ee750e5772f18b0af9ee331b43873926a108f4a5a07b495e1"
Nov 03 13:51:41.275110 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:41.275074    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:42.147232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:42.147158    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:43.040848 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:43.040809    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:43.869176 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:43.869103    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:44.715397 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:44.715365    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:45.481712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:45.481679    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:45.806605 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:45.806527    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:45.938802 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:45.938767    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:46.118281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:46.118189    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:46.290865 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:46.290832    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:46.396145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:46.396065    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"]
Nov 03 13:51:46.399699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:46.399675    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d969013e-thtzq"]
Nov 03 13:51:46.445019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:46.444992    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:47.083304 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:47.083274    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306ba9c0-e04d-42e0-92d8-59a0650b0129" path="/var/lib/kubelet/pods/306ba9c0-e04d-42e0-92d8-59a0650b0129/volumes"
Nov 03 13:51:47.529998 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:47.529963    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:51:48.768866 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:48.768835    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:48.908302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:48.908275    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:51.925947 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:51.925913    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:51:53.670027 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:51:53.669997    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:51:54.734053 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:51:54.734010    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:52:09.631964 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:09.631927    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:52:13.896577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:13.896539    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:52:17.073725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:17.073687    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:52:20.367469 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:20.367434    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:52:23.340577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:23.340548    5420 scope.go:122] "RemoveContainer" containerID="d41bd9da2616cb32a2e3fcf79ab93343a8874016a05f0608787bd28d5dcfdb8c"
Nov 03 13:52:24.730127 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:52:24.730087    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:52:32.582093 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:32.582058    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:52:36.576353 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:36.576311    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:52:36.576982 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:36.576950    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:52:36.710381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:36.710345    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:52:36.710542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:36.710413    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:52:54.729718 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:52:54.729679    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:52:59.523411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:52:59.523332    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:53:00.249302 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:00.249248    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:53:00.575293 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:00.559965    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:53:01.943725 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:01.943688    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:53:05.081238 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:05.079452    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:53:20.415678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:20.415645    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:53:24.729384 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:53:24.729347    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:53:24.781727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:24.781701    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:53:43.344073 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:43.344042    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:53:51.306427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.303518    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"]
Nov 03 13:53:51.306427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.304458    5420 cpu_manager.go:414] "containerMap: removing container" podUID="306ba9c0-e04d-42e0-92d8-59a0650b0129" containerName="revoke-keys"
Nov 03 13:53:51.306427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.304479    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306ba9c0-e04d-42e0-92d8-59a0650b0129" containerName="revoke-keys"
Nov 03 13:53:51.306427 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.304655    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="306ba9c0-e04d-42e0-92d8-59a0650b0129" containerName="revoke-keys"
Nov 03 13:53:51.309369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.308708    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.313598 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.313571    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:53:51.313831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.313619    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:53:51.313831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.313694    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 13:53:51.314339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.314319    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"]
Nov 03 13:53:51.314822 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.314797    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:53:51.344905 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.344869    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5f4ff47-vhp5z\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.345089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.344933    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqdz\" (UniqueName: \"kubernetes.io/projected/634bd2ad-b135-44f6-a1ae-2542ba29be07-kube-api-access-bwqdz\") pod \"maas-api-revoke-sub-b5f4ff47-vhp5z\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.345089 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.345054    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5f4ff47-vhp5z\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.445931 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.445899    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5f4ff47-vhp5z\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.446132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.445944    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqdz\" (UniqueName: \"kubernetes.io/projected/634bd2ad-b135-44f6-a1ae-2542ba29be07-kube-api-access-bwqdz\") pod \"maas-api-revoke-sub-b5f4ff47-vhp5z\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.446132 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.446064    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5f4ff47-vhp5z\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.446732 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.446711    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5f4ff47-vhp5z\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.448316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.448297    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5f4ff47-vhp5z\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.457348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.457320    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqdz\" (UniqueName: \"kubernetes.io/projected/634bd2ad-b135-44f6-a1ae-2542ba29be07-kube-api-access-bwqdz\") pod \"maas-api-revoke-sub-b5f4ff47-vhp5z\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.626744 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.626657    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:51.902167 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:51.902141    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"]
Nov 03 13:53:51.904567 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:53:51.904533    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634bd2ad_b135_44f6_a1ae_2542ba29be07.slice/crio-5c689f4b42c12d6a725ddd25b665714e9a6a28b4d9e19486417d2293ba459592 WatchSource:0}: Error finding container 5c689f4b42c12d6a725ddd25b665714e9a6a28b4d9e19486417d2293ba459592: Status 404 returned error can't find the container with id 5c689f4b42c12d6a725ddd25b665714e9a6a28b4d9e19486417d2293ba459592
Nov 03 13:53:52.643812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:52.643712    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z" event={"ID":"634bd2ad-b135-44f6-a1ae-2542ba29be07","Type":"ContainerStarted","Data":"30daae31f6e01f2ebe8d09cf578b060b96edc7f7fd80f38b62963aa0e01d4826"}
Nov 03 13:53:52.643812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:52.643767    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z" event={"ID":"634bd2ad-b135-44f6-a1ae-2542ba29be07","Type":"ContainerStarted","Data":"5c689f4b42c12d6a725ddd25b665714e9a6a28b4d9e19486417d2293ba459592"}
Nov 03 13:53:52.662816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:52.662767    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z" podStartSLOduration=1.662752909 podStartE2EDuration="1.662752909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:52.660925657 +0000 UTC m=+2673.738690683" watchObservedRunningTime="2026-11-03 13:53:52.662752909 +0000 UTC m=+2673.740517935"
Nov 03 13:53:53.650342 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:53.650302    5420 generic.go:356] "Generic (PLEG): container finished" podID="634bd2ad-b135-44f6-a1ae-2542ba29be07" containerID="30daae31f6e01f2ebe8d09cf578b060b96edc7f7fd80f38b62963aa0e01d4826" exitCode=0
Nov 03 13:53:53.650741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:53.650385    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z" event={"ID":"634bd2ad-b135-44f6-a1ae-2542ba29be07","Type":"ContainerDied","Data":"30daae31f6e01f2ebe8d09cf578b060b96edc7f7fd80f38b62963aa0e01d4826"}
Nov 03 13:53:54.741698 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:53:54.741656    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:53:54.794429 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.794404    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:54.874057 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.874022    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-openshift-ca\") pod \"634bd2ad-b135-44f6-a1ae-2542ba29be07\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") "
Nov 03 13:53:54.874281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.874078    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-xks-ca\") pod \"634bd2ad-b135-44f6-a1ae-2542ba29be07\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") "
Nov 03 13:53:54.874281 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.874106    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634bd2ad-b135-44f6-a1ae-2542ba29be07-kube-api-access-bwqdz\" (UniqueName: \"kubernetes.io/projected/634bd2ad-b135-44f6-a1ae-2542ba29be07-kube-api-access-bwqdz\") pod \"634bd2ad-b135-44f6-a1ae-2542ba29be07\" (UID: \"634bd2ad-b135-44f6-a1ae-2542ba29be07\") "
Nov 03 13:53:54.874452 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.874432    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-openshift-ca" pod "634bd2ad-b135-44f6-a1ae-2542ba29be07" (UID: "634bd2ad-b135-44f6-a1ae-2542ba29be07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:53:54.876317 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.876288    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-xks-ca" pod "634bd2ad-b135-44f6-a1ae-2542ba29be07" (UID: "634bd2ad-b135-44f6-a1ae-2542ba29be07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:53:54.876433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.876345    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634bd2ad-b135-44f6-a1ae-2542ba29be07-kube-api-access-bwqdz" pod "634bd2ad-b135-44f6-a1ae-2542ba29be07" (UID: "634bd2ad-b135-44f6-a1ae-2542ba29be07"). InnerVolumeSpecName "kube-api-access-bwqdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:53:54.975082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.975048    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:53:54.975082 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.975081    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/634bd2ad-b135-44f6-a1ae-2542ba29be07-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:53:54.975314 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:54.975092    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwqdz\" (UniqueName: \"kubernetes.io/projected/634bd2ad-b135-44f6-a1ae-2542ba29be07-kube-api-access-bwqdz\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:53:55.662439 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:55.662405    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"
Nov 03 13:53:55.662643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:55.662411    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z" event={"ID":"634bd2ad-b135-44f6-a1ae-2542ba29be07","Type":"ContainerDied","Data":"5c689f4b42c12d6a725ddd25b665714e9a6a28b4d9e19486417d2293ba459592"}
Nov 03 13:53:55.662643 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:53:55.662581    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c689f4b42c12d6a725ddd25b665714e9a6a28b4d9e19486417d2293ba459592"
Nov 03 13:54:02.343466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:02.343432    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:54:03.174063 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:03.174029    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:54:05.009760 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:05.009729    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:54:05.887358 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:05.887305    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5v6m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:54:05.887686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:05.887639    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5v6m6" podUID="d62992ac-fb2f-447d-b8e4-83328832a8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:54:07.059529 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.059491    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"]
Nov 03 13:54:07.060121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.060097    5420 cpu_manager.go:414] "containerMap: removing container" podUID="634bd2ad-b135-44f6-a1ae-2542ba29be07" containerName="revoke-keys"
Nov 03 13:54:07.060172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.060124    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634bd2ad-b135-44f6-a1ae-2542ba29be07" containerName="revoke-keys"
Nov 03 13:54:07.060291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.060276    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="634bd2ad-b135-44f6-a1ae-2542ba29be07" containerName="revoke-keys"
Nov 03 13:54:07.064179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.064157    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.076787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.071584    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:54:07.076787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.071704    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:54:07.076787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.071790    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:54:07.076787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.071886    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 13:54:07.076787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.075333    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"]
Nov 03 13:54:07.110493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.110462    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a4c374f-277pq\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.110676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.110504    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxmh7\" (UniqueName: \"kubernetes.io/projected/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-kube-api-access-rxmh7\") pod \"maas-api-revoke-sub-5a4c374f-277pq\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.110676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.110619    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a4c374f-277pq\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.211602 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.211567    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a4c374f-277pq\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.211778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.211611    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxmh7\" (UniqueName: \"kubernetes.io/projected/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-kube-api-access-rxmh7\") pod \"maas-api-revoke-sub-5a4c374f-277pq\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.211778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.211640    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a4c374f-277pq\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.212334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.212312    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a4c374f-277pq\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.214034 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.214016    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a4c374f-277pq\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.220763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.220719    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxmh7\" (UniqueName: \"kubernetes.io/projected/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-kube-api-access-rxmh7\") pod \"maas-api-revoke-sub-5a4c374f-277pq\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.257121 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.257094    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:54:07.393273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.393161    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:07.675286 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.675260    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"]
Nov 03 13:54:07.677405 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:54:07.677371    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a219014_6ac3_45b2_bb0e_a3e73baf1c84.slice/crio-3ac32e4835f02ddb1d0f4cd3f6efcfbb9b42bf55effec9c23d9f1d2a2770df68 WatchSource:0}: Error finding container 3ac32e4835f02ddb1d0f4cd3f6efcfbb9b42bf55effec9c23d9f1d2a2770df68: Status 404 returned error can't find the container with id 3ac32e4835f02ddb1d0f4cd3f6efcfbb9b42bf55effec9c23d9f1d2a2770df68
Nov 03 13:54:07.723303 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:07.723258    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq" event={"ID":"0a219014-6ac3-45b2-bb0e-a3e73baf1c84","Type":"ContainerStarted","Data":"3ac32e4835f02ddb1d0f4cd3f6efcfbb9b42bf55effec9c23d9f1d2a2770df68"}
Nov 03 13:54:08.756240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:08.753094    5420 generic.go:356] "Generic (PLEG): container finished" podID="0a219014-6ac3-45b2-bb0e-a3e73baf1c84" containerID="03c219459990c071b77a7d07f671827fceb94f4f200a844dda7b741a76084bf9" exitCode=0
Nov 03 13:54:08.756240 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:08.753175    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq" event={"ID":"0a219014-6ac3-45b2-bb0e-a3e73baf1c84","Type":"ContainerDied","Data":"03c219459990c071b77a7d07f671827fceb94f4f200a844dda7b741a76084bf9"}
Nov 03 13:54:10.137235 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.124321    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:10.298857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.298824    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-xks-ca\") pod \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") "
Nov 03 13:54:10.299075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.298889    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-kube-api-access-rxmh7\" (UniqueName: \"kubernetes.io/projected/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-kube-api-access-rxmh7\") pod \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") "
Nov 03 13:54:10.299075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.298938    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-openshift-ca\") pod \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\" (UID: \"0a219014-6ac3-45b2-bb0e-a3e73baf1c84\") "
Nov 03 13:54:10.299381 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.299357    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-openshift-ca" pod "0a219014-6ac3-45b2-bb0e-a3e73baf1c84" (UID: "0a219014-6ac3-45b2-bb0e-a3e73baf1c84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:54:10.301009 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.300987    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-xks-ca" pod "0a219014-6ac3-45b2-bb0e-a3e73baf1c84" (UID: "0a219014-6ac3-45b2-bb0e-a3e73baf1c84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:54:10.301133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.301058    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-kube-api-access-rxmh7" pod "0a219014-6ac3-45b2-bb0e-a3e73baf1c84" (UID: "0a219014-6ac3-45b2-bb0e-a3e73baf1c84"). InnerVolumeSpecName "kube-api-access-rxmh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:54:10.400322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.400242    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:54:10.400322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.400278    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxmh7\" (UniqueName: \"kubernetes.io/projected/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-kube-api-access-rxmh7\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:54:10.400322 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.400290    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a219014-6ac3-45b2-bb0e-a3e73baf1c84-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:54:10.766831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.766806    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"
Nov 03 13:54:10.766997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.766844    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq" event={"ID":"0a219014-6ac3-45b2-bb0e-a3e73baf1c84","Type":"ContainerDied","Data":"3ac32e4835f02ddb1d0f4cd3f6efcfbb9b42bf55effec9c23d9f1d2a2770df68"}
Nov 03 13:54:10.766997 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:10.766892    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ac32e4835f02ddb1d0f4cd3f6efcfbb9b42bf55effec9c23d9f1d2a2770df68"
Nov 03 13:54:11.246919 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:11.246886    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:54:19.682403 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:19.682372    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:54:19.693967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:19.693944    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:54:20.466206 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:20.466174    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:54:21.754066 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.754031    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"]
Nov 03 13:54:21.754639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.754620    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0a219014-6ac3-45b2-bb0e-a3e73baf1c84" containerName="revoke-keys"
Nov 03 13:54:21.754695 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.754643    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a219014-6ac3-45b2-bb0e-a3e73baf1c84" containerName="revoke-keys"
Nov 03 13:54:21.754793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.754780    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a219014-6ac3-45b2-bb0e-a3e73baf1c84" containerName="revoke-keys"
Nov 03 13:54:21.758343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.758323    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.765586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.765303    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6a354560-kserve-self-signed-certs\""
Nov 03 13:54:21.767928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.767906    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"]
Nov 03 13:54:21.799881 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.799855    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-home\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.800059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.799899    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-tmp-dir\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.800059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.799932    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dck4c\" (UniqueName: \"kubernetes.io/projected/f74ecca9-533c-495a-bc84-e417b8c6835b-kube-api-access-dck4c\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.800201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.800077    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74ecca9-533c-495a-bc84-e417b8c6835b-tls-certs\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.800201 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.800175    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-model-cache\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.800351 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.800270    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-dshm\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.856542 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.856509    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:54:21.901641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.901605    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74ecca9-533c-495a-bc84-e417b8c6835b-tls-certs\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.901907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.901704    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-model-cache\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.901907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.901740    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-dshm\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.901907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.901800    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-home\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.901907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.901840    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-tmp-dir\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.901907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.901874    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dck4c\" (UniqueName: \"kubernetes.io/projected/f74ecca9-533c-495a-bc84-e417b8c6835b-kube-api-access-dck4c\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.902482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.902379    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-tmp-dir\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.902482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.902415    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-model-cache\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.902482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.902433    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-home\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.904264 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.904246    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74ecca9-533c-495a-bc84-e417b8c6835b-tls-certs\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.904508 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.904489    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-dshm\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:21.910637 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:21.910610    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dck4c\" (UniqueName: \"kubernetes.io/projected/f74ecca9-533c-495a-bc84-e417b8c6835b-kube-api-access-dck4c\") pod \"e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") " pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:22.071503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:22.071425    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:22.344091 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:22.344059    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"]
Nov 03 13:54:22.346221 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:54:22.346183    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf74ecca9_533c_495a_bc84_e417b8c6835b.slice/crio-b8239344187c25e248099a9e2bb3aadfe8178ea83abe374b82ab617fc3f9948c WatchSource:0}: Error finding container b8239344187c25e248099a9e2bb3aadfe8178ea83abe374b82ab617fc3f9948c: Status 404 returned error can't find the container with id b8239344187c25e248099a9e2bb3aadfe8178ea83abe374b82ab617fc3f9948c
Nov 03 13:54:22.824337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:22.824299    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q" event={"ID":"f74ecca9-533c-495a-bc84-e417b8c6835b","Type":"ContainerStarted","Data":"afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1"}
Nov 03 13:54:22.824337 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:22.824341    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q" event={"ID":"f74ecca9-533c-495a-bc84-e417b8c6835b","Type":"ContainerStarted","Data":"b8239344187c25e248099a9e2bb3aadfe8178ea83abe374b82ab617fc3f9948c"}
Nov 03 13:54:22.844100 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:22.844050    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q" podStartSLOduration=1.844036907 podStartE2EDuration="1.844036907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:54:22.842860301 +0000 UTC m=+2703.920625328" watchObservedRunningTime="2026-11-03 13:54:22.844036907 +0000 UTC m=+2703.921801927"
Nov 03 13:54:23.071657 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:23.071625    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:23.226417 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:23.226380    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:54:24.801843 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:54:24.801784    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:54:27.505138 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:27.505060    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:54:32.071812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:32.071776    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:32.084330 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:32.084307    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:32.921553 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:32.921525    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:54:38.173763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.173723    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"]
Nov 03 13:54:38.178252 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.178228    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.187189 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.187162    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6a354560-kserve-self-signed-certs\""
Nov 03 13:54:38.189109 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.189082    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"]
Nov 03 13:54:38.278678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.278646    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:54:38.307184 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.307142    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f24a0ca-cb8b-47de-bff5-70b31e321455-tls-certs\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.307610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.307578    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-dshm\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.307823 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.307774    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxr22\" (UniqueName: \"kubernetes.io/projected/9f24a0ca-cb8b-47de-bff5-70b31e321455-kube-api-access-nxr22\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.307932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.307827    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-model-cache\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.307932 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.307851    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-home\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.308058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.308040    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-tmp-dir\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.408915 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.408887    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxr22\" (UniqueName: \"kubernetes.io/projected/9f24a0ca-cb8b-47de-bff5-70b31e321455-kube-api-access-nxr22\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.409102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.408939    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-model-cache\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.409102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.408963    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-home\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.409102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.409002    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-tmp-dir\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.409102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.409064    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f24a0ca-cb8b-47de-bff5-70b31e321455-tls-certs\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.409102 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.409098    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-dshm\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.409492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.409462    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-tmp-dir\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.409492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.409480    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-model-cache\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.409663 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.409540    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-home\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.411180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.411159    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-dshm\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.411461 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.411440    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f24a0ca-cb8b-47de-bff5-70b31e321455-tls-certs\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.417290 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.417267    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxr22\" (UniqueName: \"kubernetes.io/projected/9f24a0ca-cb8b-47de-bff5-70b31e321455-kube-api-access-nxr22\") pod \"e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") " pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.491319 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.491288    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:38.764229 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.764153    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"]
Nov 03 13:54:38.766818 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:54:38.766793    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f24a0ca_cb8b_47de_bff5_70b31e321455.slice/crio-413472bd7610c5fc78de2d0d3dd3e38a838de867198b98ba23906e821c1e426d WatchSource:0}: Error finding container 413472bd7610c5fc78de2d0d3dd3e38a838de867198b98ba23906e821c1e426d: Status 404 returned error can't find the container with id 413472bd7610c5fc78de2d0d3dd3e38a838de867198b98ba23906e821c1e426d
Nov 03 13:54:38.949084 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.949040    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7" event={"ID":"9f24a0ca-cb8b-47de-bff5-70b31e321455","Type":"ContainerStarted","Data":"7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd"}
Nov 03 13:54:38.949311 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.949097    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7" event={"ID":"9f24a0ca-cb8b-47de-bff5-70b31e321455","Type":"ContainerStarted","Data":"413472bd7610c5fc78de2d0d3dd3e38a838de867198b98ba23906e821c1e426d"}
Nov 03 13:54:38.969320 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:38.969272    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7" podStartSLOduration=0.969258232 podStartE2EDuration="969.258232ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:54:38.967109767 +0000 UTC m=+2720.044874794" watchObservedRunningTime="2026-11-03 13:54:38.969258232 +0000 UTC m=+2720.047023258"
Nov 03 13:54:39.491613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:39.491568    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:39.947662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:39.947589    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:54:48.491939 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:48.491902    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:48.504292 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:48.504267    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:49.039481 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:49.039450    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:54:54.730227 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:54:54.730168    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:54:56.265147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:54:56.265109    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:55:06.530974 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:06.530941    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"]
Nov 03 13:55:06.626895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:06.626844    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:07.366625 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:07.366588    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"]
Nov 03 13:55:07.483425 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:07.483390    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:07.937080 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:07.937045    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:55:14.067764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:14.067728    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:55:19.944259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:19.944227    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:55:20.683841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:20.683810    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:21.553590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:21.553524    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q" podUID="f74ecca9-533c-495a-bc84-e417b8c6835b" containerName="main" containerID="cri-o://afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1" gracePeriod=45
Nov 03 13:55:21.573401 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:21.573371    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:55:21.573618 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:21.573589    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:55:21.710348 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:21.710325    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:55:21.710514 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:21.710495    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:55:22.009477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.009449    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:55:22.096903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.096818    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-home\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-home\") pod \"f74ecca9-533c-495a-bc84-e417b8c6835b\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") "
Nov 03 13:55:22.096903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.096866    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-tmp-dir\") pod \"f74ecca9-533c-495a-bc84-e417b8c6835b\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") "
Nov 03 13:55:22.096903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.096900    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74ecca9-533c-495a-bc84-e417b8c6835b-kube-api-access-dck4c\" (UniqueName: \"kubernetes.io/projected/f74ecca9-533c-495a-bc84-e417b8c6835b-kube-api-access-dck4c\") pod \"f74ecca9-533c-495a-bc84-e417b8c6835b\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") "
Nov 03 13:55:22.097250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.096947    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74ecca9-533c-495a-bc84-e417b8c6835b-tls-certs\" (UniqueName: \"kubernetes.io/secret/f74ecca9-533c-495a-bc84-e417b8c6835b-tls-certs\") pod \"f74ecca9-533c-495a-bc84-e417b8c6835b\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") "
Nov 03 13:55:22.097250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.096986    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-model-cache\") pod \"f74ecca9-533c-495a-bc84-e417b8c6835b\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") "
Nov 03 13:55:22.097250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.097172    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-dshm\") pod \"f74ecca9-533c-495a-bc84-e417b8c6835b\" (UID: \"f74ecca9-533c-495a-bc84-e417b8c6835b\") "
Nov 03 13:55:22.097250 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.097189    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-tmp-dir" pod "f74ecca9-533c-495a-bc84-e417b8c6835b" (UID: "f74ecca9-533c-495a-bc84-e417b8c6835b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:55:22.097597 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.097575    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.097758 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.097739    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-model-cache" pod "f74ecca9-533c-495a-bc84-e417b8c6835b" (UID: "f74ecca9-533c-495a-bc84-e417b8c6835b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:55:22.097945 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.097920    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-home" pod "f74ecca9-533c-495a-bc84-e417b8c6835b" (UID: "f74ecca9-533c-495a-bc84-e417b8c6835b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:55:22.099883 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.099839    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74ecca9-533c-495a-bc84-e417b8c6835b-kube-api-access-dck4c" pod "f74ecca9-533c-495a-bc84-e417b8c6835b" (UID: "f74ecca9-533c-495a-bc84-e417b8c6835b"). InnerVolumeSpecName "kube-api-access-dck4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:55:22.100143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.100097    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74ecca9-533c-495a-bc84-e417b8c6835b-tls-certs" pod "f74ecca9-533c-495a-bc84-e417b8c6835b" (UID: "f74ecca9-533c-495a-bc84-e417b8c6835b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:55:22.100268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.100239    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-dshm" pod "f74ecca9-533c-495a-bc84-e417b8c6835b" (UID: "f74ecca9-533c-495a-bc84-e417b8c6835b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:55:22.198244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.198205    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74ecca9-533c-495a-bc84-e417b8c6835b-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.198244 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.198244    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.198414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.198253    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.198414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.198262    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74ecca9-533c-495a-bc84-e417b8c6835b-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.198414 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.198271    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dck4c\" (UniqueName: \"kubernetes.io/projected/f74ecca9-533c-495a-bc84-e417b8c6835b-kube-api-access-dck4c\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.256239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.256187    5420 generic.go:356] "Generic (PLEG): container finished" podID="f74ecca9-533c-495a-bc84-e417b8c6835b" containerID="afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1" exitCode=0
Nov 03 13:55:22.256376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.256265    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"
Nov 03 13:55:22.256376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.256259    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q" event={"ID":"f74ecca9-533c-495a-bc84-e417b8c6835b","Type":"ContainerDied","Data":"afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1"}
Nov 03 13:55:22.256475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.256381    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q" event={"ID":"f74ecca9-533c-495a-bc84-e417b8c6835b","Type":"ContainerDied","Data":"b8239344187c25e248099a9e2bb3aadfe8178ea83abe374b82ab617fc3f9948c"}
Nov 03 13:55:22.256475 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.256398    5420 scope.go:122] "RemoveContainer" containerID="afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1"
Nov 03 13:55:22.266268 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.266246    5420 scope.go:122] "RemoveContainer" containerID="afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1"
Nov 03 13:55:22.266535 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:55:22.266514    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1\": container with ID starting with afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1 not found: ID does not exist" containerID="afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1"
Nov 03 13:55:22.266613 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.266551    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1"} err="failed to get container status \"afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1\": rpc error: code = NotFound desc = could not find container \"afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1\": container with ID starting with afd82ffe60c6b1c63283cb8d10f2fcf4fc60b971e0fd1ec6bc61acb80949c7d1 not found: ID does not exist"
Nov 03 13:55:22.279126 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.279103    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"]
Nov 03 13:55:22.284120 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.284099    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q"]
Nov 03 13:55:22.386583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.386490    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7" podUID="9f24a0ca-cb8b-47de-bff5-70b31e321455" containerName="main" containerID="cri-o://7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd" gracePeriod=45
Nov 03 13:55:22.644934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.644881    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:55:22.701287 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701255    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f24a0ca-cb8b-47de-bff5-70b31e321455-kube-api-access-nxr22\" (UniqueName: \"kubernetes.io/projected/9f24a0ca-cb8b-47de-bff5-70b31e321455-kube-api-access-nxr22\") pod \"9f24a0ca-cb8b-47de-bff5-70b31e321455\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") "
Nov 03 13:55:22.701470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701300    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-dshm\") pod \"9f24a0ca-cb8b-47de-bff5-70b31e321455\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") "
Nov 03 13:55:22.701470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701406    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f24a0ca-cb8b-47de-bff5-70b31e321455-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f24a0ca-cb8b-47de-bff5-70b31e321455-tls-certs\") pod \"9f24a0ca-cb8b-47de-bff5-70b31e321455\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") "
Nov 03 13:55:22.701470 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701444    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-home\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-home\") pod \"9f24a0ca-cb8b-47de-bff5-70b31e321455\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") "
Nov 03 13:55:22.701647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701476    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-tmp-dir\") pod \"9f24a0ca-cb8b-47de-bff5-70b31e321455\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") "
Nov 03 13:55:22.701647 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701535    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-model-cache\") pod \"9f24a0ca-cb8b-47de-bff5-70b31e321455\" (UID: \"9f24a0ca-cb8b-47de-bff5-70b31e321455\") "
Nov 03 13:55:22.701756 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701707    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-home" pod "9f24a0ca-cb8b-47de-bff5-70b31e321455" (UID: "9f24a0ca-cb8b-47de-bff5-70b31e321455"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:55:22.701811 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701777    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-tmp-dir" pod "9f24a0ca-cb8b-47de-bff5-70b31e321455" (UID: "9f24a0ca-cb8b-47de-bff5-70b31e321455"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:55:22.701938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701905    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.701938 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701935    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.702131 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.701931    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-model-cache" pod "9f24a0ca-cb8b-47de-bff5-70b31e321455" (UID: "9f24a0ca-cb8b-47de-bff5-70b31e321455"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:55:22.703466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.703439    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f24a0ca-cb8b-47de-bff5-70b31e321455-tls-certs" pod "9f24a0ca-cb8b-47de-bff5-70b31e321455" (UID: "9f24a0ca-cb8b-47de-bff5-70b31e321455"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:55:22.703543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.703479    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f24a0ca-cb8b-47de-bff5-70b31e321455-kube-api-access-nxr22" pod "9f24a0ca-cb8b-47de-bff5-70b31e321455" (UID: "9f24a0ca-cb8b-47de-bff5-70b31e321455"). InnerVolumeSpecName "kube-api-access-nxr22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:55:22.703543 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.703513    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-dshm" pod "9f24a0ca-cb8b-47de-bff5-70b31e321455" (UID: "9f24a0ca-cb8b-47de-bff5-70b31e321455"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:55:22.803143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.803117    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.803143 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.803139    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxr22\" (UniqueName: \"kubernetes.io/projected/9f24a0ca-cb8b-47de-bff5-70b31e321455-kube-api-access-nxr22\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.803340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.803150    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f24a0ca-cb8b-47de-bff5-70b31e321455-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.803340 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.803158    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f24a0ca-cb8b-47de-bff5-70b31e321455-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:55:22.913115 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:22.913007    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-6a354560-kserve-58644b47cc-shk6q" podUID="f74ecca9-533c-495a-bc84-e417b8c6835b" containerName="main" probeResult="failure" output="Get \"https://10.134.1.20:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:55:23.084789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.084751    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74ecca9-533c-495a-bc84-e417b8c6835b" path="/var/lib/kubelet/pods/f74ecca9-533c-495a-bc84-e417b8c6835b/volumes"
Nov 03 13:55:23.263567 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.263529    5420 generic.go:356] "Generic (PLEG): container finished" podID="9f24a0ca-cb8b-47de-bff5-70b31e321455" containerID="7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd" exitCode=0
Nov 03 13:55:23.263771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.263579    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"
Nov 03 13:55:23.263771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.263621    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7" event={"ID":"9f24a0ca-cb8b-47de-bff5-70b31e321455","Type":"ContainerDied","Data":"7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd"}
Nov 03 13:55:23.263771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.263671    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7" event={"ID":"9f24a0ca-cb8b-47de-bff5-70b31e321455","Type":"ContainerDied","Data":"413472bd7610c5fc78de2d0d3dd3e38a838de867198b98ba23906e821c1e426d"}
Nov 03 13:55:23.263771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.263698    5420 scope.go:122] "RemoveContainer" containerID="7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd"
Nov 03 13:55:23.275316 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.275296    5420 scope.go:122] "RemoveContainer" containerID="7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd"
Nov 03 13:55:23.275654 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:55:23.275625    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd\": container with ID starting with 7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd not found: ID does not exist" containerID="7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd"
Nov 03 13:55:23.275786 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.275666    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd"} err="failed to get container status \"7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd\": rpc error: code = NotFound desc = could not find container \"7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd\": container with ID starting with 7cc6b57e2e27a77861daf6041a9dca942b5624cbeba9bca769c38c42347061bd not found: ID does not exist"
Nov 03 13:55:23.283894 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.283867    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"]
Nov 03 13:55:23.286591 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:23.286569    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6a354560-kserve-69687b969c-bgqb7"]
Nov 03 13:55:24.796478 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:55:24.796424    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:55:24.988477 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:24.988443    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:55:25.107595 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:25.107512    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f24a0ca-cb8b-47de-bff5-70b31e321455" path="/var/lib/kubelet/pods/9f24a0ca-cb8b-47de-bff5-70b31e321455/volumes"
Nov 03 13:55:38.162486 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:38.162450    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:55:38.791948 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:38.791866    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:39.171279 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:39.171178    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:39.315075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:39.315038    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:42.105684 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:42.105650    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:42.256014 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:42.255985    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:43.389405 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:43.389372    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:43.549478 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:43.549445    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:44.353859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:44.353824    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:45.248306 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:45.248275    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:55:45.256686 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:45.256661    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:46.169701 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:46.169664    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:46.978590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:46.976707    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:47.879615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:47.879557    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:48.691366 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:48.691312    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:49.489830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:49.489797    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:50.369313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:50.369243    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:51.158773 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:51.158738    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:51.960579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:51.960547    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:52.790616 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:52.790587    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:53.686005 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:53.685977    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:54.464291 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:54.464249    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:54.730332 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:55:54.730257    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:55:55.243743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:55.243707    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:55.964722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:55.964692    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:56.803363 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:56.803331    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:57.173149 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:57.173069    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:57.309064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:57.309030    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:57.536503 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:57.536471    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:57.675971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:57.675937    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:58.507843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:58.507791    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:59.082950 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:59.082919    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:59.238119 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:59.238083    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:55:59.684821 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:55:59.684787    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:00.591727 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:00.591694    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:01.744482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:01.744447    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:02.633000 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:02.632967    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:03.547525 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:03.547496    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:04.273326 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:04.273294    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:05.146771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:05.146736    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:05.881365 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:05.881331    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:06.758764 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:06.758730    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:07.564742 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:07.564702    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:08.456445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:08.456409    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:09.219903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:09.219863    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:10.147453 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:10.147412    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:10.865271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:10.865237    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:11.856644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:11.856605    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:12.770106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:12.770075    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:13.662113 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:13.662079    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:14.465002 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:14.464968    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:15.354382 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:15.354349    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:16.077534 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:16.077505    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:17.069273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:17.069233    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:17.855609 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:17.855574    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:18.756462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:18.756423    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:18.991979 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:18.991946    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:19.449179 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:19.449147    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:19.598518 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:19.598475    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:19.735012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:19.734970    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:20.525153 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:20.525117    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:20.719678 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:20.719651    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:21.374935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:21.374902    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:22.397474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:22.397438    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:22.601432 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:22.601395    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:23.476343 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.476306    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:23.675935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.675892    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:23.679433 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.679409    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"]
Nov 03 13:56:23.679863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.679846    5420 cpu_manager.go:414] "containerMap: removing container" podUID="9f24a0ca-cb8b-47de-bff5-70b31e321455" containerName="main"
Nov 03 13:56:23.679935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.679864    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f24a0ca-cb8b-47de-bff5-70b31e321455" containerName="main"
Nov 03 13:56:23.679935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.679895    5420 cpu_manager.go:414] "containerMap: removing container" podUID="f74ecca9-533c-495a-bc84-e417b8c6835b" containerName="main"
Nov 03 13:56:23.679935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.679901    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74ecca9-533c-495a-bc84-e417b8c6835b" containerName="main"
Nov 03 13:56:23.680058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.679959    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f24a0ca-cb8b-47de-bff5-70b31e321455" containerName="main"
Nov 03 13:56:23.680058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.679973    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74ecca9-533c-495a-bc84-e417b8c6835b" containerName="main"
Nov 03 13:56:23.682955 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.682940    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:23.687694 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.687674    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4d1115\""
Nov 03 13:56:23.696789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.696766    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"]
Nov 03 13:56:23.722497 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.722462    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"]
Nov 03 13:56:23.726787 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.726725    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:23.730794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.730774    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4d1115-dockercfg-s5mkj\""
Nov 03 13:56:23.736893 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.736857    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"]
Nov 03 13:56:23.795759 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.795717    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bvrv\" (UniqueName: \"kubernetes.io/projected/3105672a-a485-4fd3-a7ed-2cbbd0b03283-kube-api-access-5bvrv\") pod \"payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh\" (UID: \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:23.800404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.795925    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3105672a-a485-4fd3-a7ed-2cbbd0b03283-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh\" (UID: \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:23.833586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.833538    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:23.896933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.896894    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvrv\" (UniqueName: \"kubernetes.io/projected/3105672a-a485-4fd3-a7ed-2cbbd0b03283-kube-api-access-5bvrv\") pod \"payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh\" (UID: \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:23.897135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.896984    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3105672a-a485-4fd3-a7ed-2cbbd0b03283-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh\" (UID: \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:23.897135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.897020    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97wr\" (UniqueName: \"kubernetes.io/projected/4d56807f-2098-4c2b-9fd9-09672e21ea54-kube-api-access-x97wr\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn\" (UID: \"4d56807f-2098-4c2b-9fd9-09672e21ea54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:23.897135 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.897057    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56807f-2098-4c2b-9fd9-09672e21ea54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn\" (UID: \"4d56807f-2098-4c2b-9fd9-09672e21ea54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:23.897722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.897702    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3105672a-a485-4fd3-a7ed-2cbbd0b03283-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh\" (UID: \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:23.906278 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.906253    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvrv\" (UniqueName: \"kubernetes.io/projected/3105672a-a485-4fd3-a7ed-2cbbd0b03283-kube-api-access-5bvrv\") pod \"payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh\" (UID: \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:23.995501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.995413    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:23.997886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.997860    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97wr\" (UniqueName: \"kubernetes.io/projected/4d56807f-2098-4c2b-9fd9-09672e21ea54-kube-api-access-x97wr\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn\" (UID: \"4d56807f-2098-4c2b-9fd9-09672e21ea54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:23.997960 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.997905    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56807f-2098-4c2b-9fd9-09672e21ea54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn\" (UID: \"4d56807f-2098-4c2b-9fd9-09672e21ea54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:23.998753 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:23.998628    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56807f-2098-4c2b-9fd9-09672e21ea54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn\" (UID: \"4d56807f-2098-4c2b-9fd9-09672e21ea54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:24.009521 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.009492    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97wr\" (UniqueName: \"kubernetes.io/projected/4d56807f-2098-4c2b-9fd9-09672e21ea54-kube-api-access-x97wr\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn\" (UID: \"4d56807f-2098-4c2b-9fd9-09672e21ea54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:24.040895 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.040867    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:24.276884 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.276857    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"]
Nov 03 13:56:24.278963 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:56:24.278933    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3105672a_a485_4fd3_a7ed_2cbbd0b03283.slice/crio-87d18779f55e322cf4a14eb1c11330e5d5dcc21665e8357acee4724fa1ceaf54 WatchSource:0}: Error finding container 87d18779f55e322cf4a14eb1c11330e5d5dcc21665e8357acee4724fa1ceaf54: Status 404 returned error can't find the container with id 87d18779f55e322cf4a14eb1c11330e5d5dcc21665e8357acee4724fa1ceaf54
Nov 03 13:56:24.334354 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.334327    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"]
Nov 03 13:56:24.336951 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:56:24.336924    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d56807f_2098_4c2b_9fd9_09672e21ea54.slice/crio-431c22db08f6b0e97c50389b03455e95a86cc7234c4fd4577c7bd10f0d4ea47f WatchSource:0}: Error finding container 431c22db08f6b0e97c50389b03455e95a86cc7234c4fd4577c7bd10f0d4ea47f: Status 404 returned error can't find the container with id 431c22db08f6b0e97c50389b03455e95a86cc7234c4fd4577c7bd10f0d4ea47f
Nov 03 13:56:24.569687 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.569581    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:24.595566 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.595523    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh" event={"ID":"3105672a-a485-4fd3-a7ed-2cbbd0b03283","Type":"ContainerStarted","Data":"8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8"}
Nov 03 13:56:24.595767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.595727    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:24.595851 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.595764    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh" event={"ID":"3105672a-a485-4fd3-a7ed-2cbbd0b03283","Type":"ContainerStarted","Data":"87d18779f55e322cf4a14eb1c11330e5d5dcc21665e8357acee4724fa1ceaf54"}
Nov 03 13:56:24.597104 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.597077    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn" event={"ID":"4d56807f-2098-4c2b-9fd9-09672e21ea54","Type":"ContainerStarted","Data":"95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93"}
Nov 03 13:56:24.597259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.597115    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn" event={"ID":"4d56807f-2098-4c2b-9fd9-09672e21ea54","Type":"ContainerStarted","Data":"431c22db08f6b0e97c50389b03455e95a86cc7234c4fd4577c7bd10f0d4ea47f"}
Nov 03 13:56:24.597259 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.597206    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:24.621141 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.621097    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh" podStartSLOduration=1.621085465 podStartE2EDuration="1.621085465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:56:24.619480166 +0000 UTC m=+2825.697245192" watchObservedRunningTime="2026-11-03 13:56:24.621085465 +0000 UTC m=+2825.698850504"
Nov 03 13:56:24.638509 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:24.638463    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn" podStartSLOduration=1.6384504880000001 podStartE2EDuration="1.638450488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:56:24.637844889 +0000 UTC m=+2825.715609914" watchObservedRunningTime="2026-11-03 13:56:24.638450488 +0000 UTC m=+2825.716215514"
Nov 03 13:56:24.730653 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:56:24.730617    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:56:25.364413 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:25.364381    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:26.375399 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:26.375364    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:26.509912 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:26.509884    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:26.654171 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:26.654093    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:27.167641 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:27.167611    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:27.973812 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:27.973784    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:28.747771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:28.747729    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:56:28.870386 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:28.870346    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:29.682916 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:29.682885    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:30.481648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:30.480835    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:31.484906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:31.484874    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:32.379232 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:32.379112    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:33.283886 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:33.283851    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:34.261203 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:34.261171    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:35.118967 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:35.118928    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:35.605284 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:35.605255    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:35.605441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:35.605409    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:36.066524 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:36.066487    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:36.953676 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:36.953643    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:37.342738 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:37.342652    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"]
Nov 03 13:56:37.354934 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:37.354907    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"]
Nov 03 13:56:37.473784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:37.473752    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:37.610362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:37.610274    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:37.748028 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:37.747988    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:37.885610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:37.885540    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:38.686462 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:38.686429    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:39.703994 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:39.703957    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:40.385735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:40.385705    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:40.537577 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:40.537546    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:40.975361 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:40.975315    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:41.981771 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:41.981722    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:42.343075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.342993    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh" podUID="3105672a-a485-4fd3-a7ed-2cbbd0b03283" containerName="payload-processing" containerID="cri-o://8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8" gracePeriod=25
Nov 03 13:56:42.355619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.355528    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn" podUID="4d56807f-2098-4c2b-9fd9-09672e21ea54" containerName="payload-pre-processing" containerID="cri-o://95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93" gracePeriod=25
Nov 03 13:56:42.592574 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.592545    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:42.624202 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.624181    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:42.677784 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.677751    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3105672a-a485-4fd3-a7ed-2cbbd0b03283-kube-api-access-5bvrv\" (UniqueName: \"kubernetes.io/projected/3105672a-a485-4fd3-a7ed-2cbbd0b03283-kube-api-access-5bvrv\") pod \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\" (UID: \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\") "
Nov 03 13:56:42.677971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.677895    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d56807f-2098-4c2b-9fd9-09672e21ea54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56807f-2098-4c2b-9fd9-09672e21ea54-plugins-config-volume\") pod \"4d56807f-2098-4c2b-9fd9-09672e21ea54\" (UID: \"4d56807f-2098-4c2b-9fd9-09672e21ea54\") "
Nov 03 13:56:42.677971 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.677950    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3105672a-a485-4fd3-a7ed-2cbbd0b03283-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3105672a-a485-4fd3-a7ed-2cbbd0b03283-plugins-config-volume\") pod \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\" (UID: \"3105672a-a485-4fd3-a7ed-2cbbd0b03283\") "
Nov 03 13:56:42.678106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.678005    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d56807f-2098-4c2b-9fd9-09672e21ea54-kube-api-access-x97wr\" (UniqueName: \"kubernetes.io/projected/4d56807f-2098-4c2b-9fd9-09672e21ea54-kube-api-access-x97wr\") pod \"4d56807f-2098-4c2b-9fd9-09672e21ea54\" (UID: \"4d56807f-2098-4c2b-9fd9-09672e21ea54\") "
Nov 03 13:56:42.678346 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.678317    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d56807f-2098-4c2b-9fd9-09672e21ea54-plugins-config-volume" pod "4d56807f-2098-4c2b-9fd9-09672e21ea54" (UID: "4d56807f-2098-4c2b-9fd9-09672e21ea54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:56:42.678430 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.678344    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3105672a-a485-4fd3-a7ed-2cbbd0b03283-plugins-config-volume" pod "3105672a-a485-4fd3-a7ed-2cbbd0b03283" (UID: "3105672a-a485-4fd3-a7ed-2cbbd0b03283"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:56:42.679933 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.679904    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3105672a-a485-4fd3-a7ed-2cbbd0b03283-kube-api-access-5bvrv" pod "3105672a-a485-4fd3-a7ed-2cbbd0b03283" (UID: "3105672a-a485-4fd3-a7ed-2cbbd0b03283"). InnerVolumeSpecName "kube-api-access-5bvrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:56:42.680058 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.679992    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d56807f-2098-4c2b-9fd9-09672e21ea54-kube-api-access-x97wr" pod "4d56807f-2098-4c2b-9fd9-09672e21ea54" (UID: "4d56807f-2098-4c2b-9fd9-09672e21ea54"). InnerVolumeSpecName "kube-api-access-x97wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:56:42.688409 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.688382    5420 generic.go:356] "Generic (PLEG): container finished" podID="3105672a-a485-4fd3-a7ed-2cbbd0b03283" containerID="8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8" exitCode=0
Nov 03 13:56:42.688523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.688429    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"
Nov 03 13:56:42.688523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.688466    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh" event={"ID":"3105672a-a485-4fd3-a7ed-2cbbd0b03283","Type":"ContainerDied","Data":"8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8"}
Nov 03 13:56:42.688635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.688551    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh" event={"ID":"3105672a-a485-4fd3-a7ed-2cbbd0b03283","Type":"ContainerDied","Data":"87d18779f55e322cf4a14eb1c11330e5d5dcc21665e8357acee4724fa1ceaf54"}
Nov 03 13:56:42.688635 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.688578    5420 scope.go:122] "RemoveContainer" containerID="8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8"
Nov 03 13:56:42.690092 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.690070    5420 generic.go:356] "Generic (PLEG): container finished" podID="4d56807f-2098-4c2b-9fd9-09672e21ea54" containerID="95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93" exitCode=0
Nov 03 13:56:42.690198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.690113    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn" event={"ID":"4d56807f-2098-4c2b-9fd9-09672e21ea54","Type":"ContainerDied","Data":"95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93"}
Nov 03 13:56:42.690198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.690123    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"
Nov 03 13:56:42.690198 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.690154    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn" event={"ID":"4d56807f-2098-4c2b-9fd9-09672e21ea54","Type":"ContainerDied","Data":"431c22db08f6b0e97c50389b03455e95a86cc7234c4fd4577c7bd10f0d4ea47f"}
Nov 03 13:56:42.697438 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.697416    5420 scope.go:122] "RemoveContainer" containerID="8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8"
Nov 03 13:56:42.697704 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:56:42.697680    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8\": container with ID starting with 8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8 not found: ID does not exist" containerID="8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8"
Nov 03 13:56:42.697805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.697713    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8"} err="failed to get container status \"8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8\": rpc error: code = NotFound desc = could not find container \"8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8\": container with ID starting with 8784930650d356fa52fb5c253b9a1be7d247cda9b1c3a03e92ad30b05feb51b8 not found: ID does not exist"
Nov 03 13:56:42.697805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.697737    5420 scope.go:122] "RemoveContainer" containerID="95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93"
Nov 03 13:56:42.708369 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.708349    5420 scope.go:122] "RemoveContainer" containerID="95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93"
Nov 03 13:56:42.708658 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:56:42.708636    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93\": container with ID starting with 95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93 not found: ID does not exist" containerID="95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93"
Nov 03 13:56:42.708743 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.708662    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93"} err="failed to get container status \"95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93\": rpc error: code = NotFound desc = could not find container \"95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93\": container with ID starting with 95a35f23c68d367d88282acfd17f843cb1d59ab0ac542a013f973b11fe772e93 not found: ID does not exist"
Nov 03 13:56:42.718191 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.718171    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"]
Nov 03 13:56:42.725376 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.725357    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-8b4cdc999-vj6xh"]
Nov 03 13:56:42.741906 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.741876    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"]
Nov 03 13:56:42.746805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.746788    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-86bc9c8975-4xldn"]
Nov 03 13:56:42.778745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.778718    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3105672a-a485-4fd3-a7ed-2cbbd0b03283-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:56:42.778745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.778742    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x97wr\" (UniqueName: \"kubernetes.io/projected/4d56807f-2098-4c2b-9fd9-09672e21ea54-kube-api-access-x97wr\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:56:42.778968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.778754    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bvrv\" (UniqueName: \"kubernetes.io/projected/3105672a-a485-4fd3-a7ed-2cbbd0b03283-kube-api-access-5bvrv\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:56:42.778968 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:42.778763    5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56807f-2098-4c2b-9fd9-09672e21ea54-plugins-config-volume\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:56:43.015736 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:43.015696    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:43.082261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:43.082203    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3105672a-a485-4fd3-a7ed-2cbbd0b03283" path="/var/lib/kubelet/pods/3105672a-a485-4fd3-a7ed-2cbbd0b03283/volumes"
Nov 03 13:56:43.082593 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:43.082580    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d56807f-2098-4c2b-9fd9-09672e21ea54" path="/var/lib/kubelet/pods/4d56807f-2098-4c2b-9fd9-09672e21ea54/volumes"
Nov 03 13:56:43.982767 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:43.982722    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:44.877428 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:44.877392    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:45.941023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:45.940993    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:46.850464 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:46.850430    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:47.755741 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:47.755703    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:48.670373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:48.670339    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:49.662047 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:49.662011    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:50.491445 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:50.491368    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:51.478463 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:51.478419    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:52.380877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:52.380838    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:53.269944 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:53.269914    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:54.288441 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:54.288405    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:54.730349 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:56:54.730314    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:56:55.165907 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:55.165811    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:55.865877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:55.865845    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:56.834582 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:56.834549    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:57.763560 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:57.763516    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:58.457789 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:58.457754    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:56:59.470500 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:56:59.470466    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:00.375069 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.375038    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:00.854420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.854388    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"]
Nov 03 13:57:00.854830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.854771    5420 cpu_manager.go:414] "containerMap: removing container" podUID="3105672a-a485-4fd3-a7ed-2cbbd0b03283" containerName="payload-processing"
Nov 03 13:57:00.854830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.854787    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3105672a-a485-4fd3-a7ed-2cbbd0b03283" containerName="payload-processing"
Nov 03 13:57:00.854830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.854813    5420 cpu_manager.go:414] "containerMap: removing container" podUID="4d56807f-2098-4c2b-9fd9-09672e21ea54" containerName="payload-pre-processing"
Nov 03 13:57:00.854830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.854822    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d56807f-2098-4c2b-9fd9-09672e21ea54" containerName="payload-pre-processing"
Nov 03 13:57:00.855043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.854923    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3105672a-a485-4fd3-a7ed-2cbbd0b03283" containerName="payload-processing"
Nov 03 13:57:00.855043 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.854936    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d56807f-2098-4c2b-9fd9-09672e21ea54" containerName="payload-pre-processing"
Nov 03 13:57:00.858362 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.858332    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:00.869539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.869517    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d0441c\"/\"default-dockercfg-p5dhr\""
Nov 03 13:57:00.869703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.869621    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d0441c\"/\"ipp-route-d0441c-kserve-self-signed-certs\""
Nov 03 13:57:00.870372 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.870353    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"]
Nov 03 13:57:00.904981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.904944    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d0441c\"/\"openshift-service-ca.crt\""
Nov 03 13:57:00.905150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.905058    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d0441c\"/\"kube-root-ca.crt\""
Nov 03 13:57:00.959404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.959368    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-model-cache\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:00.959404 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.959403    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-tmp-dir\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:00.959619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.959423    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-dshm\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:00.959619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.959529    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-home\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:00.959619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.959548    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daca604b-0a12-4b14-b620-da7680264f17-tls-certs\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:00.959619 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:00.959584    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46st\" (UniqueName: \"kubernetes.io/projected/daca604b-0a12-4b14-b620-da7680264f17-kube-api-access-l46st\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.015012 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.014980    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:01.061299 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.061261    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-home\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.061662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.061608    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daca604b-0a12-4b14-b620-da7680264f17-tls-certs\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.061819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.061670    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-home\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.061819 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.061717    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46st\" (UniqueName: \"kubernetes.io/projected/daca604b-0a12-4b14-b620-da7680264f17-kube-api-access-l46st\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.061958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.061828    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-model-cache\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.061958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.061875    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-tmp-dir\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.061958 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.061902    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-dshm\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.062415 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.062392    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-tmp-dir\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.062579 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.062436    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-model-cache\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.064241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.064192    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-dshm\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.064778 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.064754    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daca604b-0a12-4b14-b620-da7680264f17-tls-certs\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.071411 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.071378    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46st\" (UniqueName: \"kubernetes.io/projected/daca604b-0a12-4b14-b620-da7680264f17-kube-api-access-l46st\") pod \"ipp-route-d0441c-kserve-77868bb9b5-tbhpd\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") " pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.173830 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.173756    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:01.437583 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.437507    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:01.453768 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.453745    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"]
Nov 03 13:57:01.455654 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:57:01.455628    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaca604b_0a12_4b14_b620_da7680264f17.slice/crio-9c0ac49751f663da0f4ff364524c23043b85536ee271414f6b2d5d8d263b1a2b WatchSource:0}: Error finding container 9c0ac49751f663da0f4ff364524c23043b85536ee271414f6b2d5d8d263b1a2b: Status 404 returned error can't find the container with id 9c0ac49751f663da0f4ff364524c23043b85536ee271414f6b2d5d8d263b1a2b
Nov 03 13:57:01.784511 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.784476    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd" event={"ID":"daca604b-0a12-4b14-b620-da7680264f17","Type":"ContainerStarted","Data":"9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7"}
Nov 03 13:57:01.784715 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.784526    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd" event={"ID":"daca604b-0a12-4b14-b620-da7680264f17","Type":"ContainerStarted","Data":"9c0ac49751f663da0f4ff364524c23043b85536ee271414f6b2d5d8d263b1a2b"}
Nov 03 13:57:01.805107 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:01.805054    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd" podStartSLOduration=1.8050411579999999 podStartE2EDuration="1.805041158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:57:01.80230813 +0000 UTC m=+2862.880073155" watchObservedRunningTime="2026-11-03 13:57:01.805041158 +0000 UTC m=+2862.882806184"
Nov 03 13:57:02.174111 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:02.174028    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:02.401554 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:02.401523    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:02.659902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:02.659866    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:57:03.575943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:03.575909    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:04.378474 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:04.378445    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:05.463809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:05.463773    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:06.360872 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:06.360819    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:07.167273 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:07.167146    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:08.089339 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:08.089309    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:09.054696 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:09.054665    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:10.075580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:10.075547    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:11.168644 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:11.168605    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:11.174539 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:11.174516    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:11.187599 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:11.187577    5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:11.842797 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:11.842767    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:12.161174 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:12.161093    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:12.977267 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:12.977194    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:13.955805 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:13.955772    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:14.958075 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:14.958026    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:15.771400 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:15.771361    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:16.704867 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:16.704825    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:17.661055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:17.661022    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:18.592809 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:18.592771    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:19.476816 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:19.476785    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:20.368735 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:20.368699    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:20.779065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:20.779013    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:21.179666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:21.179547    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:22.138262 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:22.138199    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:23.062501 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:23.062464    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:23.876125 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:23.876076    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:24.730258 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:57:24.730197    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:57:24.825661 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:24.825611    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:25.745965 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:25.745926    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:26.551194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:26.551113    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:27.468416 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:27.468384    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:28.364903 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:28.364871    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:29.058482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:29.058443    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:57:29.277813 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:29.277773    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:30.252055 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:30.252018    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:31.161506 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:31.161356    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:31.958331 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:31.958290    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:32.866434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:32.866403    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:33.763068 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:33.763035    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:34.568033 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:34.567992    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:35.470984 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:35.470946    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:36.365660 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:36.365627    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:36.638545 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:36.638480    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:37.162254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:37.162194    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:38.071807 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:38.071764    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:38.958857 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:38.958817    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:39.734840 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:39.734803    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:40.747713 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:40.747674    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:41.563853 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:41.563817    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"]
Nov 03 13:57:41.658067 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:41.658025    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:41.798606 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:41.798569    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:42.471237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:42.471187    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:43.381832 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:43.381800    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:44.272367 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:44.272333    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:45.060586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:45.060530    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:46.062901 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:46.062859    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:47.065271 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:47.065234    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:47.862505 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:47.862471    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:48.890022 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:48.889986    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:50.763897 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:50.763847    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:50.897494 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:50.897466    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:51.468482 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:51.468450    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:52.464898 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:52.464866    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:53.175730 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:53.175694    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:54.061384 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:54.061343    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:54.730348 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:57:54.730316    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:57:55.059675 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:55.059539    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:55.958634 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:55.958597    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:56.584029 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.583975    5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd" podUID="daca604b-0a12-4b14-b620-da7680264f17" containerName="main" containerID="cri-o://9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7" gracePeriod=45
Nov 03 13:57:56.840792 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.840724    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:56.868954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.868883    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:56.912860 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.912825    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-dshm\") pod \"daca604b-0a12-4b14-b620-da7680264f17\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") "
Nov 03 13:57:56.913061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.912880    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daca604b-0a12-4b14-b620-da7680264f17-kube-api-access-l46st\" (UniqueName: \"kubernetes.io/projected/daca604b-0a12-4b14-b620-da7680264f17-kube-api-access-l46st\") pod \"daca604b-0a12-4b14-b620-da7680264f17\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") "
Nov 03 13:57:56.913061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.912929    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daca604b-0a12-4b14-b620-da7680264f17-tls-certs\" (UniqueName: \"kubernetes.io/secret/daca604b-0a12-4b14-b620-da7680264f17-tls-certs\") pod \"daca604b-0a12-4b14-b620-da7680264f17\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") "
Nov 03 13:57:56.913061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.912964    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-model-cache\") pod \"daca604b-0a12-4b14-b620-da7680264f17\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") "
Nov 03 13:57:56.913061 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.913007    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-home\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-home\") pod \"daca604b-0a12-4b14-b620-da7680264f17\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") "
Nov 03 13:57:56.913313 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.913135    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-tmp-dir\") pod \"daca604b-0a12-4b14-b620-da7680264f17\" (UID: \"daca604b-0a12-4b14-b620-da7680264f17\") "
Nov 03 13:57:56.913440 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.913397    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-home" pod "daca604b-0a12-4b14-b620-da7680264f17" (UID: "daca604b-0a12-4b14-b620-da7680264f17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:57:56.913621 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.913464    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-model-cache" pod "daca604b-0a12-4b14-b620-da7680264f17" (UID: "daca604b-0a12-4b14-b620-da7680264f17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:57:56.913719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.913639    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-tmp-dir" pod "daca604b-0a12-4b14-b620-da7680264f17" (UID: "daca604b-0a12-4b14-b620-da7680264f17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:57:56.915145 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.915120    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-dshm" pod "daca604b-0a12-4b14-b620-da7680264f17" (UID: "daca604b-0a12-4b14-b620-da7680264f17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:57:56.915493 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.915474    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daca604b-0a12-4b14-b620-da7680264f17-tls-certs" pod "daca604b-0a12-4b14-b620-da7680264f17" (UID: "daca604b-0a12-4b14-b620-da7680264f17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:57:56.915586 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:56.915498    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daca604b-0a12-4b14-b620-da7680264f17-kube-api-access-l46st" pod "daca604b-0a12-4b14-b620-da7680264f17" (UID: "daca604b-0a12-4b14-b620-da7680264f17"). InnerVolumeSpecName "kube-api-access-l46st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:57:57.014276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.014245    5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-home\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:57:57.014276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.014271    5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-tmp-dir\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:57:57.014276 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.014280    5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-dshm\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:57:57.014719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.014289    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l46st\" (UniqueName: \"kubernetes.io/projected/daca604b-0a12-4b14-b620-da7680264f17-kube-api-access-l46st\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:57:57.014719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.014299    5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daca604b-0a12-4b14-b620-da7680264f17-tls-certs\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:57:57.014719 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.014307    5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daca604b-0a12-4b14-b620-da7680264f17-model-cache\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:57:57.055639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.055612    5420 generic.go:356] "Generic (PLEG): container finished" podID="daca604b-0a12-4b14-b620-da7680264f17" containerID="9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7" exitCode=0
Nov 03 13:57:57.055808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.055657    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"
Nov 03 13:57:57.055808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.055701    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd" event={"ID":"daca604b-0a12-4b14-b620-da7680264f17","Type":"ContainerDied","Data":"9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7"}
Nov 03 13:57:57.055808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.055741    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd" event={"ID":"daca604b-0a12-4b14-b620-da7680264f17","Type":"ContainerDied","Data":"9c0ac49751f663da0f4ff364524c23043b85536ee271414f6b2d5d8d263b1a2b"}
Nov 03 13:57:57.055808 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.055761    5420 scope.go:122] "RemoveContainer" containerID="9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7"
Nov 03 13:57:57.065420 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.065404    5420 scope.go:122] "RemoveContainer" containerID="9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7"
Nov 03 13:57:57.065762 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:57:57.065738    5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7\": container with ID starting with 9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7 not found: ID does not exist" containerID="9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7"
Nov 03 13:57:57.065875 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.065770    5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7"} err="failed to get container status \"9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7\": rpc error: code = NotFound desc = could not find container \"9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7\": container with ID starting with 9ce3d3e24b9d9030db4138a6b80adeb9f46d289e7168b0239bf9534468f512b7 not found: ID does not exist"
Nov 03 13:57:57.082044 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.082025    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"]
Nov 03 13:57:57.082355 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.082338    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd"]
Nov 03 13:57:57.755133 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.755095    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:57.831603 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:57.831565    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-d0441c/ipp-route-d0441c-kserve-77868bb9b5-tbhpd" podUID="daca604b-0a12-4b14-b620-da7680264f17" containerName="main" probeResult="failure" output="Get \"https://10.134.1.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:57:58.573180 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:58.573139    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:57:59.082737 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:59.082694    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daca604b-0a12-4b14-b620-da7680264f17" path="/var/lib/kubelet/pods/daca604b-0a12-4b14-b620-da7680264f17/volumes"
Nov 03 13:57:59.549527 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:57:59.549429    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:00.461561 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:00.461513    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:01.274920 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:01.274852    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:02.173078 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:02.173012    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:03.187682 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:03.187637    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:03.556648 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:03.556617    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:03.964645 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:03.964614    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:04.969418 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:04.969385    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:05.928246 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:05.927908    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:06.669655 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:06.669621    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:07.561454 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:07.561424    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:08.451334 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:08.451279    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:09.258449 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:09.258411    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:10.185589 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:10.185555    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:11.068254 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:11.068221    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:11.947877 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:11.947840    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:12.867019 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:12.866985    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:13.518666 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:13.518638    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:13.657698 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:13.657666    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:13.795658 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:13.795575    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:14.550709 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:14.550657    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:15.483059 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:15.483023    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:16.367981 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:16.367944    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:17.156946 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:17.156903    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:18.081479 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:18.081449    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:18.230699 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:18.230658    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:19.244237 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.244194    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"]
Nov 03 13:58:19.244624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.244584    5420 cpu_manager.go:414] "containerMap: removing container" podUID="daca604b-0a12-4b14-b620-da7680264f17" containerName="main"
Nov 03 13:58:19.244624 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.244595    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daca604b-0a12-4b14-b620-da7680264f17" containerName="main"
Nov 03 13:58:19.244703 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.244653    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="daca604b-0a12-4b14-b620-da7680264f17" containerName="main"
Nov 03 13:58:19.247626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.247605    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.259117 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.259094    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 13:58:19.259296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.259140    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 13:58:19.259296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.259165    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:58:19.259296 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.259191    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 13:58:19.263466 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.263444    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"]
Nov 03 13:58:19.275943 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.275920    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:19.323859 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.323836    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh7ph\" (UniqueName: \"kubernetes.io/projected/0b36ea3b-f623-4973-ab0d-974dda115989-kube-api-access-sh7ph\") pod \"maas-api-revoke-sub-5e7dd2a6-npmnn\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.324001 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.323869    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7dd2a6-npmnn\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.324072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.323999    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7dd2a6-npmnn\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.415437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.415404    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:19.424662 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.424619    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7dd2a6-npmnn\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.424825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.424708    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh7ph\" (UniqueName: \"kubernetes.io/projected/0b36ea3b-f623-4973-ab0d-974dda115989-kube-api-access-sh7ph\") pod \"maas-api-revoke-sub-5e7dd2a6-npmnn\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.424825 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.424750    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7dd2a6-npmnn\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.425385 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.425363    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7dd2a6-npmnn\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.427064 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.427044    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7dd2a6-npmnn\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.435199 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.435175    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh7ph\" (UniqueName: \"kubernetes.io/projected/0b36ea3b-f623-4973-ab0d-974dda115989-kube-api-access-sh7ph\") pod \"maas-api-revoke-sub-5e7dd2a6-npmnn\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.560712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.560631    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:19.840519 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:19.840489    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"]
Nov 03 13:58:19.842782 ip-10-0-139-248 kubenswrapper[5420]: W1103 13:58:19.842750    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b36ea3b_f623_4973_ab0d_974dda115989.slice/crio-70385f7346805884b8bb5eb201b137eb8149b40a668d2b4af897bac94e42f05f WatchSource:0}: Error finding container 70385f7346805884b8bb5eb201b137eb8149b40a668d2b4af897bac94e42f05f: Status 404 returned error can't find the container with id 70385f7346805884b8bb5eb201b137eb8149b40a668d2b4af897bac94e42f05f
Nov 03 13:58:20.175672 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:20.175646    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn" event={"ID":"0b36ea3b-f623-4973-ab0d-974dda115989","Type":"ContainerStarted","Data":"45d17878f96c8c35f2735d2bc58e3ae0cdb45dc68d2f41acb63ad9fb8624a5a4"}
Nov 03 13:58:20.175793 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:20.175694    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn" event={"ID":"0b36ea3b-f623-4973-ab0d-974dda115989","Type":"ContainerStarted","Data":"70385f7346805884b8bb5eb201b137eb8149b40a668d2b4af897bac94e42f05f"}
Nov 03 13:58:20.194989 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:20.194947    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn" podStartSLOduration=1.19493318 podStartE2EDuration="1.19493318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:58:20.193476863 +0000 UTC m=+2941.271241891" watchObservedRunningTime="2026-11-03 13:58:20.19493318 +0000 UTC m=+2941.272698205"
Nov 03 13:58:20.273318 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:20.273280    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:20.466437 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:20.466405    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:58:20.844492 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:20.844453    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:21.181957 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:21.181874    5420 generic.go:356] "Generic (PLEG): container finished" podID="0b36ea3b-f623-4973-ab0d-974dda115989" containerID="45d17878f96c8c35f2735d2bc58e3ae0cdb45dc68d2f41acb63ad9fb8624a5a4" exitCode=0
Nov 03 13:58:21.182105 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:21.181973    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn" event={"ID":"0b36ea3b-f623-4973-ab0d-974dda115989","Type":"ContainerDied","Data":"45d17878f96c8c35f2735d2bc58e3ae0cdb45dc68d2f41acb63ad9fb8624a5a4"}
Nov 03 13:58:21.253632 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:21.253602    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:22.267754 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.267716    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:22.323373 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.323349    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:22.452531 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.452501    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-xks-ca\") pod \"0b36ea3b-f623-4973-ab0d-974dda115989\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") "
Nov 03 13:58:22.452712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.452546    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b36ea3b-f623-4973-ab0d-974dda115989-kube-api-access-sh7ph\" (UniqueName: \"kubernetes.io/projected/0b36ea3b-f623-4973-ab0d-974dda115989-kube-api-access-sh7ph\") pod \"0b36ea3b-f623-4973-ab0d-974dda115989\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") "
Nov 03 13:58:22.452712 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.452631    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-openshift-ca\") pod \"0b36ea3b-f623-4973-ab0d-974dda115989\" (UID: \"0b36ea3b-f623-4973-ab0d-974dda115989\") "
Nov 03 13:58:22.453035 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.453003    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-openshift-ca" pod "0b36ea3b-f623-4973-ab0d-974dda115989" (UID: "0b36ea3b-f623-4973-ab0d-974dda115989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:58:22.454718 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.454691    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b36ea3b-f623-4973-ab0d-974dda115989-kube-api-access-sh7ph" pod "0b36ea3b-f623-4973-ab0d-974dda115989" (UID: "0b36ea3b-f623-4973-ab0d-974dda115989"). InnerVolumeSpecName "kube-api-access-sh7ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:58:22.454810 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.454758    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-xks-ca" pod "0b36ea3b-f623-4973-ab0d-974dda115989" (UID: "0b36ea3b-f623-4973-ab0d-974dda115989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:58:22.553722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.553689    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:58:22.553722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.553718    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh7ph\" (UniqueName: \"kubernetes.io/projected/0b36ea3b-f623-4973-ab0d-974dda115989-kube-api-access-sh7ph\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:58:22.553722 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:22.553730    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b36ea3b-f623-4973-ab0d-974dda115989-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 13:58:23.164359 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:23.164324    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:23.193562 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:23.193533    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"
Nov 03 13:58:23.193763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:23.193569    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn" event={"ID":"0b36ea3b-f623-4973-ab0d-974dda115989","Type":"ContainerDied","Data":"70385f7346805884b8bb5eb201b137eb8149b40a668d2b4af897bac94e42f05f"}
Nov 03 13:58:23.193763 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:23.193616    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70385f7346805884b8bb5eb201b137eb8149b40a668d2b4af897bac94e42f05f"
Nov 03 13:58:23.424601 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:23.424524    5420 scope.go:122] "RemoveContainer" containerID="09edd8e4fe296643eda16b356057ff5dede99b58be8dc06795021e3551b41dd1"
Nov 03 13:58:24.078952 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:24.078924    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:24.729318 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:58:24.729282    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:58:25.060928 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:25.060832    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:25.236480 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:25.236437    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:25.871241 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:25.871183    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:26.766090 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:26.766046    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:27.553008 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:27.552966    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:28.458615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:28.458538    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:29.365072 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:29.365036    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:30.249941 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:30.249903    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:31.063065 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:31.063027    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:31.195831 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:31.195800    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:32.048159 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:32.048128    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:32.786769 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:32.786732    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:33.664782 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:33.664742    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:34.261282 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:34.261244    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:34.387408 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:34.387375    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:34.754247 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:34.754188    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:35.674935 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:35.674905    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:35.859610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:35.859578    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:58:36.678863 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:36.678640    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:37.224844 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:37.224806    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:37.384843 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:37.384811    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:37.742610 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:37.742575    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:37.890626 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:37.890581    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:38.195258 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:38.195142    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:38.689523 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:38.689482    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:39.666434 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:39.666402    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:40.480639 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:40.480586    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:41.357172 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:41.357143    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:42.161023 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:42.160991    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:43.052927 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:43.052888    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:43.853876 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:43.853837    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:44.300841 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:44.300808    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:44.757794 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:44.757756    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:45.478016 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:45.477972    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:46.680280 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:46.680244    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:47.595194 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:47.595154    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:48.441158 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:48.441122    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:49.315239 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:49.315180    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:50.243580 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:50.243549    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:50.371458 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:50.371428    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:50.920536 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:50.920504    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:51.053590 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:51.053553    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:52.354691 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:52.354661    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:58:54.730830 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:58:54.730796    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:58:55.035833 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:55.035743    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"]
Nov 03 13:58:55.038745 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:55.038718    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f4ff47-vhp5z"]
Nov 03 13:58:55.082222 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:55.082185    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="634bd2ad-b135-44f6-a1ae-2542ba29be07" path="/var/lib/kubelet/pods/634bd2ad-b135-44f6-a1ae-2542ba29be07/volumes"
Nov 03 13:58:55.899654 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:55.899619    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:56.038207 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:56.038174    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:56.382870 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:56.382794    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:58:56.812471 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:58:56.812436    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:59:11.022150 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:11.022114    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"]
Nov 03 13:59:11.023772 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:11.023750    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a4c374f-277pq"]
Nov 03 13:59:11.082902 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:11.082869    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a219014-6ac3-45b2-bb0e-a3e73baf1c84" path="/var/lib/kubelet/pods/0a219014-6ac3-45b2-bb0e-a3e73baf1c84/volumes"
Nov 03 13:59:17.139774 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:17.139736    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:59:19.719147 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:19.719120    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:59:19.727483 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:19.727463    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 13:59:20.836261 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:20.836227    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:59:24.730015 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:59:24.729977    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 13:59:37.371913 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:37.371870    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:59:40.710106 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:40.710071    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:59:40.907615 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:40.907578    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:59:41.268954 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:41.268917    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:59:42.382169 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:42.382136    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 13:59:46.450146 ip-10-0-139-248 kubenswrapper[5420]: I1103 13:59:46.450110    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 13:59:54.729307 ip-10-0-139-248 kubenswrapper[5420]: E1103 13:59:54.729268    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:00:20.951458 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:00:20.951425    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:00:23.456945 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:00:23.456913    5420 scope.go:122] "RemoveContainer" containerID="30daae31f6e01f2ebe8d09cf578b060b96edc7f7fd80f38b62963aa0e01d4826"
Nov 03 14:00:23.465677 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:00:23.465655    5420 scope.go:122] "RemoveContainer" containerID="03c219459990c071b77a7d07f671827fceb94f4f200a844dda7b741a76084bf9"
Nov 03 14:00:24.731237 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:00:24.731184    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:00:54.731360 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:00:54.731323    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:01:18.493696 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:01:18.493656    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:01:21.004674 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:01:21.004637    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:01:21.156444 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:01:21.156397    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:01:24.730669 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:01:24.730633    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:01:27.937644 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:01:27.937608    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:01:28.355451 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:01:28.355415    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:01:28.507444 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:01:28.507408    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:01:33.408446 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:01:33.408407    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:01:42.976090 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:01:42.976050    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:01:54.730123 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:01:54.730085    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:01:56.168143 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:01:56.168104    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:02:07.071138 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:02:07.071084    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:02:20.971807 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:02:20.971774    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:02:21.054275 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:02:21.054245    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:02:21.210395 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:02:21.210357    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:02:24.729342 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:02:24.729307    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:02:31.177901 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:02:31.177865    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:02:35.316534 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:02:35.316500    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:02:51.928130 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:02:51.928092    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:02:54.735296 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:02:54.735248    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:02:55.894705 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:02:55.893617    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:02:59.446227 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:02:59.446184    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:03:10.280601 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:10.280563    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:03:21.104749 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:21.104654    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:03:21.260110 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:21.260081    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:03:23.030018 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:23.029985    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"]
Nov 03 14:03:23.033392 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:23.033367    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7dd2a6-npmnn"]
Nov 03 14:03:23.082656 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:23.082615    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b36ea3b-f623-4973-ab0d-974dda115989" path="/var/lib/kubelet/pods/0b36ea3b-f623-4973-ab0d-974dda115989/volumes"
Nov 03 14:03:24.051988 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:24.051953    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:03:24.729616 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:03:24.729581    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:03:27.773725 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:27.773529    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:03:31.329121 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:31.329086    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:03:49.369412 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:49.369382    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:03:51.710366 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:51.710329    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:03:51.710858 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:51.710396    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:03:53.968606 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:03:53.968568    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:03:54.731021 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:03:54.730991    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:04:04.768573 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:04.768539    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:04:08.447573 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:08.447540    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:04:11.338909 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:11.338878    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:04:19.753102 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:19.753074    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 14:04:19.762796 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:19.762770    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 14:04:21.154894 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:21.154858    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:04:23.513766 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:23.513739    5420 scope.go:122] "RemoveContainer" containerID="45d17878f96c8c35f2735d2bc58e3ae0cdb45dc68d2f41acb63ad9fb8624a5a4"
Nov 03 14:04:24.730196 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:04:24.730158    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:04:29.750531 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:29.750492    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:04:35.878722 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:35.878685    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:04:39.943628 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:39.943595    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:04:54.731275 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:04:54.731236    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:04:56.448890 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:04:56.448806    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:05:00.136451 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:00.136415    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:00.554274 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:00.554243    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:00.691011 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:00.690976    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:01.676321 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:01.676285    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:05:03.541542 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:03.541501    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:03.693874 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:03.693841    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:04.792752 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:04.792712    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:04.965600 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:04.965550    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:05.892039 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:05.891997    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:06.790066 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:06.790035    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:07.360716 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:07.360683    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:05:07.933924 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:07.933890    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:08.966522 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:08.966488    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:09.771299 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:09.771268    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:10.576116 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:10.576079    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:11.407468 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:11.407430    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:12.352341 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:12.352310    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:13.237153 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:13.237117    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:14.060933 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:14.060891    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:14.851893 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:14.851850    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:15.653616 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:15.653580    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:16.465478 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:16.465439    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:17.256354 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:17.256317    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:18.093837 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:18.093759    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:18.539665 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:18.539635    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:18.974902 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:18.974864    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:19.166270 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:19.166239    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:19.310730 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:19.310643    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:19.453840 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:19.453806    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:20.017192 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:20.017148    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:20.652420 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:20.652393    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:20.961917 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:20.961873    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:21.227110 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:21.227021    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:21.859727 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:21.859695    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:05:21.888096 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:21.888061    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:22.876089 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:22.876046    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:23.741498 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:23.741460    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:24.495222 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:24.495169    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:24.729708 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:05:24.729677    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:05:25.322059 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:25.322028    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:26.057105 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:26.057050    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:26.873628 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:26.873594    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:27.667593 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:27.667562    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:28.371797 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:28.371763    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:29.134753 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:29.134720    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:29.854345 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:29.854309    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:30.640086 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:30.640052    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:31.361245 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:31.361177    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:32.145232 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:32.145170    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:32.968825 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:32.968742    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:33.875927 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:33.875891    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:34.555125 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:34.555094    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:35.467592 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:35.467557    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:36.353352 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:36.353323    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:37.257589 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:37.257556    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:38.059004 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:38.058958    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:38.956509 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:38.956468    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:39.064556 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:39.064522    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:05:39.860601 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:39.860571    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:40.675178 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:40.675122    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:41.465842 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:41.464388    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:42.374237 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:42.374191    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:43.167263 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:43.167195    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:44.000402 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:44.000370    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:44.861806 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:44.861771    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:45.646271 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:45.646229    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:46.467613 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:46.467580    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:47.274765 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:47.274734    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:48.055909 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:48.055823    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:48.846814 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:48.846786    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:49.680613 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:49.680576    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:50.451766 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.451644    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:50.621324 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.621290    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"]
Nov 03 14:05:50.621802 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.621785    5420 cpu_manager.go:414] "containerMap: removing container" podUID="0b36ea3b-f623-4973-ab0d-974dda115989" containerName="revoke-keys"
Nov 03 14:05:50.621861 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.621803    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b36ea3b-f623-4973-ab0d-974dda115989" containerName="revoke-keys"
Nov 03 14:05:50.621899 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.621893    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b36ea3b-f623-4973-ab0d-974dda115989" containerName="revoke-keys"
Nov 03 14:05:50.624832 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.624817    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.631555 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.631525    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:05:50.631555 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.631545    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:05:50.631788 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.631660    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:05:50.632442 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.632417    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:05:50.634607 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.634587    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"]
Nov 03 14:05:50.686540 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.686518    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c47143d-5flwm\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.686697 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.686562    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c47143d-5flwm\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.686697 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.686590    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlj9\" (UniqueName: \"kubernetes.io/projected/e262abe2-2b93-40df-95c6-ebc79d6e5b20-kube-api-access-qwlj9\") pod \"maas-api-revoke-sub-0c47143d-5flwm\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.787786 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.787760    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlj9\" (UniqueName: \"kubernetes.io/projected/e262abe2-2b93-40df-95c6-ebc79d6e5b20-kube-api-access-qwlj9\") pod \"maas-api-revoke-sub-0c47143d-5flwm\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.787918 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.787826    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c47143d-5flwm\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.787918 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.787877    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c47143d-5flwm\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.788472 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.788455    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c47143d-5flwm\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.789974 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.789954    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c47143d-5flwm\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.796444 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.796420    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlj9\" (UniqueName: \"kubernetes.io/projected/e262abe2-2b93-40df-95c6-ebc79d6e5b20-kube-api-access-qwlj9\") pod \"maas-api-revoke-sub-0c47143d-5flwm\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:50.941070 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:50.941041    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:51.159723 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:51.159620    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:51.214443 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:51.214408    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"]
Nov 03 14:05:51.218878 ip-10-0-139-248 kubenswrapper[5420]: W1103 14:05:51.218849    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode262abe2_2b93_40df_95c6_ebc79d6e5b20.slice/crio-6dc3f78d38822b83245da32175c21dd907e028586e4ea081cbc75d581ee462f7 WatchSource:0}: Error finding container 6dc3f78d38822b83245da32175c21dd907e028586e4ea081cbc75d581ee462f7: Status 404 returned error can't find the container with id 6dc3f78d38822b83245da32175c21dd907e028586e4ea081cbc75d581ee462f7
Nov 03 14:05:51.519487 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:51.519458    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm" event={"ID":"e262abe2-2b93-40df-95c6-ebc79d6e5b20","Type":"ContainerStarted","Data":"2be39d018493e53ef558d8088e5aad4dc7c437e54d284db0742a2d818f22cebb"}
Nov 03 14:05:51.519813 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:51.519505    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm" event={"ID":"e262abe2-2b93-40df-95c6-ebc79d6e5b20","Type":"ContainerStarted","Data":"6dc3f78d38822b83245da32175c21dd907e028586e4ea081cbc75d581ee462f7"}
Nov 03 14:05:51.538292 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:51.538247    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm" podStartSLOduration=1.53823212 podStartE2EDuration="1.53823212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:05:51.536160364 +0000 UTC m=+3392.613925396" watchObservedRunningTime="2026-11-03 14:05:51.53823212 +0000 UTC m=+3392.615997137"
Nov 03 14:05:51.945482 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:51.945446    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:52.526998 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:52.526961    5420 generic.go:356] "Generic (PLEG): container finished" podID="e262abe2-2b93-40df-95c6-ebc79d6e5b20" containerID="2be39d018493e53ef558d8088e5aad4dc7c437e54d284db0742a2d818f22cebb" exitCode=0
Nov 03 14:05:52.527422 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:52.527034    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm" event={"ID":"e262abe2-2b93-40df-95c6-ebc79d6e5b20","Type":"ContainerDied","Data":"2be39d018493e53ef558d8088e5aad4dc7c437e54d284db0742a2d818f22cebb"}
Nov 03 14:05:52.742145 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:52.742117    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:53.454473 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.454442    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:53.667399 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.667376    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:53.717154 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.717076    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-openshift-ca\") pod \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") "
Nov 03 14:05:53.717316 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.717267    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e262abe2-2b93-40df-95c6-ebc79d6e5b20-kube-api-access-qwlj9\" (UniqueName: \"kubernetes.io/projected/e262abe2-2b93-40df-95c6-ebc79d6e5b20-kube-api-access-qwlj9\") pod \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") "
Nov 03 14:05:53.717316 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.717299    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-xks-ca\") pod \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\" (UID: \"e262abe2-2b93-40df-95c6-ebc79d6e5b20\") "
Nov 03 14:05:53.717442 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.717424    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-openshift-ca" pod "e262abe2-2b93-40df-95c6-ebc79d6e5b20" (UID: "e262abe2-2b93-40df-95c6-ebc79d6e5b20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:05:53.717666 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.717649    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:05:53.719439 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.719416    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e262abe2-2b93-40df-95c6-ebc79d6e5b20-kube-api-access-qwlj9" pod "e262abe2-2b93-40df-95c6-ebc79d6e5b20" (UID: "e262abe2-2b93-40df-95c6-ebc79d6e5b20"). InnerVolumeSpecName "kube-api-access-qwlj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:05:53.719525 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.719434    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-xks-ca" pod "e262abe2-2b93-40df-95c6-ebc79d6e5b20" (UID: "e262abe2-2b93-40df-95c6-ebc79d6e5b20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:05:53.818955 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.818927    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwlj9\" (UniqueName: \"kubernetes.io/projected/e262abe2-2b93-40df-95c6-ebc79d6e5b20-kube-api-access-qwlj9\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:05:53.818955 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:53.818952    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e262abe2-2b93-40df-95c6-ebc79d6e5b20-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:05:54.330634 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:54.330600    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:54.537396 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:54.537365    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm" event={"ID":"e262abe2-2b93-40df-95c6-ebc79d6e5b20","Type":"ContainerDied","Data":"6dc3f78d38822b83245da32175c21dd907e028586e4ea081cbc75d581ee462f7"}
Nov 03 14:05:54.537396 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:54.537382    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"
Nov 03 14:05:54.537655 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:54.537412    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dc3f78d38822b83245da32175c21dd907e028586e4ea081cbc75d581ee462f7"
Nov 03 14:05:54.730178 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:05:54.730147    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:05:55.078550 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:55.078451    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:55.765634 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:55.765591    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:56.553859 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:56.553822    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:57.264045 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:57.264006    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:57.963096 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:57.963065    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:58.679469 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:58.679421    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:05:59.438118 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:05:59.438083    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:00.197544 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:00.197505    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:01.062415 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:01.062387    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:01.526022 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:01.525995    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:01.874760 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:01.874675    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:02.037381 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:02.037347    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:02.436877 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:02.436843    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:06:02.842258 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:02.842201    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:03.689542 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:03.689507    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:04.474549 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:04.474514    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:05.273041 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:05.273002    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:06.037347 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:06.037321    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:06.620719 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:06.620676    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:06.911027 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:06.910948    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:07.681419 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:07.681384    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:08.267774 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:08.267744    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:08.413601 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:08.413565    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:08.635825 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:08.635742    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:09.244155 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:09.244120    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:10.840510 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:10.840480    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:06:14.005583 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:14.005548    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:14.150221 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:14.150174    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:21.282029 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:21.281996    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:24.730275 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:06:24.730192    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:06:39.470518 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:39.470418    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:39.624191 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:39.624141    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:40.663715 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:40.663680    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:40.811554 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:40.811521    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:41.606051 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:41.606017    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:42.348427 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:42.348388    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:42.746416 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:42.746377    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:06:43.156142 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:43.156016    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:43.380510 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:43.380455    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:44.017520 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:44.017487    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:44.160397 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:44.160333    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:44.666581 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:44.666546    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:45.576196 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:45.576151    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:45.770469 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:45.770436    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:06:46.521292 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:46.521236    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:47.244827 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:47.244790    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:47.952742 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:47.952710    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:48.751514 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:48.751473    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:49.293128 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.293100    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"]
Nov 03 14:06:49.293523 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.293509    5420 cpu_manager.go:414] "containerMap: removing container" podUID="e262abe2-2b93-40df-95c6-ebc79d6e5b20" containerName="revoke-keys"
Nov 03 14:06:49.293575 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.293525    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e262abe2-2b93-40df-95c6-ebc79d6e5b20" containerName="revoke-keys"
Nov 03 14:06:49.293615 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.293585    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e262abe2-2b93-40df-95c6-ebc79d6e5b20" containerName="revoke-keys"
Nov 03 14:06:49.296583 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.296567    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.303046 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.302970    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:06:49.304393 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.304378    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:06:49.304473 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.304398    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:06:49.304473 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.304404    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\""
Nov 03 14:06:49.310728 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.310706    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"]
Nov 03 14:06:49.456617 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.456581    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdgn\" (UniqueName: \"kubernetes.io/projected/825fb029-66ae-482f-b77f-34695cd1d2c0-kube-api-access-skdgn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.456839 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.456717    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.456839 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.456797    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.483180 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.483146    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:49.558283 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.558179    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdgn\" (UniqueName: \"kubernetes.io/projected/825fb029-66ae-482f-b77f-34695cd1d2c0-kube-api-access-skdgn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.558283 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.558275    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.558496 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.558322    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.558907 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.558884    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.560648 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.560626    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.566741 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.566722    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdgn\" (UniqueName: \"kubernetes.io/projected/825fb029-66ae-482f-b77f-34695cd1d2c0-kube-api-access-skdgn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.609362 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.609339    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:49.884263 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:49.884237    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"]
Nov 03 14:06:49.886105 ip-10-0-139-248 kubenswrapper[5420]: W1103 14:06:49.886070    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825fb029_66ae_482f_b77f_34695cd1d2c0.slice/crio-88d9dc85d417635d3091efb7ebbfc0475aefb462a452bdef2388b08c04302cc4 WatchSource:0}: Error finding container 88d9dc85d417635d3091efb7ebbfc0475aefb462a452bdef2388b08c04302cc4: Status 404 returned error can't find the container with id 88d9dc85d417635d3091efb7ebbfc0475aefb462a452bdef2388b08c04302cc4
Nov 03 14:06:50.335946 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:50.335912    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:50.597111 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:50.597019    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:50.738986 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:50.738961    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:50.809092 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:50.809062    5420 generic.go:356] "Generic (PLEG): container finished" podID="825fb029-66ae-482f-b77f-34695cd1d2c0" containerID="00eaaa12f8e2890afab675c744c567c0a88ebbae9c294653cd6211f73c554cd8" exitCode=0
Nov 03 14:06:50.809303 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:50.809160    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv" event={"ID":"825fb029-66ae-482f-b77f-34695cd1d2c0","Type":"ContainerDied","Data":"00eaaa12f8e2890afab675c744c567c0a88ebbae9c294653cd6211f73c554cd8"}
Nov 03 14:06:50.809303 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:50.809221    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv" event={"ID":"825fb029-66ae-482f-b77f-34695cd1d2c0","Type":"ContainerStarted","Data":"88d9dc85d417635d3091efb7ebbfc0475aefb462a452bdef2388b08c04302cc4"}
Nov 03 14:06:51.155127 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:51.155093    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:51.902189 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:51.902161    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:51.960830 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:51.960803    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:52.083476 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.083404    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825fb029-66ae-482f-b77f-34695cd1d2c0-kube-api-access-skdgn\" (UniqueName: \"kubernetes.io/projected/825fb029-66ae-482f-b77f-34695cd1d2c0-kube-api-access-skdgn\") pod \"825fb029-66ae-482f-b77f-34695cd1d2c0\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") "
Nov 03 14:06:52.083476 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.083468    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-openshift-ca\") pod \"825fb029-66ae-482f-b77f-34695cd1d2c0\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") "
Nov 03 14:06:52.083672 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.083522    5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-xks-ca\") pod \"825fb029-66ae-482f-b77f-34695cd1d2c0\" (UID: \"825fb029-66ae-482f-b77f-34695cd1d2c0\") "
Nov 03 14:06:52.084105 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.084077    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-openshift-ca" pod "825fb029-66ae-482f-b77f-34695cd1d2c0" (UID: "825fb029-66ae-482f-b77f-34695cd1d2c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:06:52.086533 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.086496    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-xks-ca" pod "825fb029-66ae-482f-b77f-34695cd1d2c0" (UID: "825fb029-66ae-482f-b77f-34695cd1d2c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:06:52.092572 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.092548    5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825fb029-66ae-482f-b77f-34695cd1d2c0-kube-api-access-skdgn" pod "825fb029-66ae-482f-b77f-34695cd1d2c0" (UID: "825fb029-66ae-482f-b77f-34695cd1d2c0"). InnerVolumeSpecName "kube-api-access-skdgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:06:52.184555 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.184528    5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdgn\" (UniqueName: \"kubernetes.io/projected/825fb029-66ae-482f-b77f-34695cd1d2c0-kube-api-access-skdgn\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:06:52.184555 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.184552    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-openshift-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:06:52.184919 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.184562    5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825fb029-66ae-482f-b77f-34695cd1d2c0-maas-api-xks-ca\") on node \"ip-10-0-139-248.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 14:06:52.649228 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.649156    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:52.820626 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.820597    5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"
Nov 03 14:06:52.820830 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.820631    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv" event={"ID":"825fb029-66ae-482f-b77f-34695cd1d2c0","Type":"ContainerDied","Data":"88d9dc85d417635d3091efb7ebbfc0475aefb462a452bdef2388b08c04302cc4"}
Nov 03 14:06:52.820830 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:52.820696    5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88d9dc85d417635d3091efb7ebbfc0475aefb462a452bdef2388b08c04302cc4"
Nov 03 14:06:53.362127 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:53.362097    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:54.100904 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:54.100876    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:54.731038 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:06:54.731005    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:06:54.849716 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:54.849669    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:55.753946 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:55.753908    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:56.458780 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:56.458743    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:57.158281 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:57.158234    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:57.869055 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:57.869001    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:58.742906 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:58.742863    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:59.472038 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:59.472007    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:06:59.608696 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:06:59.608665    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:00.175362 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:00.175307    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:00.792848 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:00.792818    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:00.975792 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:00.975756    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:01.087841 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:01.087737    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"]
Nov 03 14:07:01.090583 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:01.090559    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-64455b-wdftv"]
Nov 03 14:07:01.119362 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:01.119342    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:01.437259 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:01.437130    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:02.264086 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:02.264047    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:07:03.082712 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:03.082679    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825fb029-66ae-482f-b77f-34695cd1d2c0" path="/var/lib/kubelet/pods/825fb029-66ae-482f-b77f-34695cd1d2c0/volumes"
Nov 03 14:07:14.034113 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:14.034080    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:07:19.175005 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:19.174973    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:07:21.477983 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:21.477947    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:21.581363 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:21.581325    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xs5rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:07:21.581705 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:21.581654    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xs5rk" podUID="2ae9c04b-23b3-4ea2-baf0-c8940a31044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:07:21.711381 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:21.711340    5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-tb9j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:07:21.711556 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:21.711416    5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-tb9j4" podUID="5762c366-b31f-488b-997b-f4ba1a053178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:07:24.730410 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:07:24.730376    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:07:34.370085 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:34.370052    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:07:42.606626 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:42.606590    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:42.765562 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:42.765531    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:43.510554 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.510524    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"]
Nov 03 14:07:43.510943 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.510909    5420 cpu_manager.go:414] "containerMap: removing container" podUID="825fb029-66ae-482f-b77f-34695cd1d2c0" containerName="revoke-keys"
Nov 03 14:07:43.510943 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.510923    5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825fb029-66ae-482f-b77f-34695cd1d2c0" containerName="revoke-keys"
Nov 03 14:07:43.511040 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.510977    5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="825fb029-66ae-482f-b77f-34695cd1d2c0" containerName="revoke-keys"
Nov 03 14:07:43.514099 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.514084    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.523524 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.523494    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"]
Nov 03 14:07:43.524264 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.524240    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:07:43.524396 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.524352    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:07:43.542207 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.542180    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 14:07:43.542621 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.542602    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 14:07:43.542719 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.542622    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p5qd5\""
Nov 03 14:07:43.591677 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.591653    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4d4f\" (UniqueName: \"kubernetes.io/projected/d5eca6d7-5050-4eaa-890e-905cbc197d52-kube-api-access-c4d4f\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.591813 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.591722    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5eca6d7-5050-4eaa-890e-905cbc197d52-openshift-service-ca\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.591871 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.591818    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5eca6d7-5050-4eaa-890e-905cbc197d52-metrics-certs\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.591871 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.591849    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5eca6d7-5050-4eaa-890e-905cbc197d52-maas-api-tls\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.693168 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.693140    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5eca6d7-5050-4eaa-890e-905cbc197d52-openshift-service-ca\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.693619 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.693174    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5eca6d7-5050-4eaa-890e-905cbc197d52-metrics-certs\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.693619 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.693193    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5eca6d7-5050-4eaa-890e-905cbc197d52-maas-api-tls\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.693619 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.693270    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4d4f\" (UniqueName: \"kubernetes.io/projected/d5eca6d7-5050-4eaa-890e-905cbc197d52-kube-api-access-c4d4f\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.693818 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.693799    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5eca6d7-5050-4eaa-890e-905cbc197d52-openshift-service-ca\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.695715 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.695686    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5eca6d7-5050-4eaa-890e-905cbc197d52-metrics-certs\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.695898 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.695878    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5eca6d7-5050-4eaa-890e-905cbc197d52-maas-api-tls\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.702351 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.702329    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4d4f\" (UniqueName: \"kubernetes.io/projected/d5eca6d7-5050-4eaa-890e-905cbc197d52-kube-api-access-c4d4f\") pod \"maas-api-7d6dc67969-rn9zr\" (UID: \"d5eca6d7-5050-4eaa-890e-905cbc197d52\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:43.828592 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:43.828500    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:44.107041 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:44.106963    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"]
Nov 03 14:07:44.109638 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:44.109614    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:44.111747 ip-10-0-139-248 kubenswrapper[5420]: W1103 14:07:44.111721    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5eca6d7_5050_4eaa_890e_905cbc197d52.slice/crio-1ae26b36bddd43454cb5c6617992dc560926446c987b1f468fd8344f9336636a WatchSource:0}: Error finding container 1ae26b36bddd43454cb5c6617992dc560926446c987b1f468fd8344f9336636a: Status 404 returned error can't find the container with id 1ae26b36bddd43454cb5c6617992dc560926446c987b1f468fd8344f9336636a
Nov 03 14:07:44.113003 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:44.112986    5420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 14:07:45.139080 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:45.136861    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr" event={"ID":"d5eca6d7-5050-4eaa-890e-905cbc197d52","Type":"ContainerStarted","Data":"1ae26b36bddd43454cb5c6617992dc560926446c987b1f468fd8344f9336636a"}
Nov 03 14:07:47.892315 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:47.892276    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:07:49.208399 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:49.208361    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr" event={"ID":"d5eca6d7-5050-4eaa-890e-905cbc197d52","Type":"ContainerStarted","Data":"4e3d5a787ea51918f3649522b7e54c8c3aa532317ae4fd583ed219a2fc4c1735"}
Nov 03 14:07:49.208818 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:49.208428    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:07:49.226161 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:49.226114    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr" podStartSLOduration=2.086747967 podStartE2EDuration="6.226099892s" totalImagesPullingTime="4.139351925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:49.226016077 +0000 UTC m=+3510.303781103" watchObservedRunningTime="2026-11-03 14:07:49.226099892 +0000 UTC m=+3510.303864912"
Nov 03 14:07:53.037743 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:53.037692    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:07:54.730593 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:07:54.730548    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:07:55.219323 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:07:55.219298    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-rn9zr"
Nov 03 14:08:04.546629 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:08:04.546594    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:08:04.888913 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:08:04.888836    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:08:10.845868 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:08:10.845825    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:08:21.525039 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:08:21.525009    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:08:24.730325 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:08:24.730283    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:08:30.719311 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:08:30.719279    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:08:54.730240 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:08:54.730186    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:09:03.902348 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:09:03.902293    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:09:19.785030 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:09:19.785004    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 14:09:19.795499 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:09:19.795480    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 14:09:21.567567 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:09:21.567529    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:09:24.729998 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:09:24.729960    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:09:27.502241 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:09:27.502133    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:09:35.357665 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:09:35.357627    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:09:35.511752 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:09:35.511712    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:09:46.254782 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:09:46.254748    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:09:54.730498 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:09:54.730462    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:09:57.423567 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:09:57.423523    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:10:03.214237 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:10:03.214172    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:10:21.632373 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:10:21.632320    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:10:24.729754 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:10:24.729715    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:10:54.077363 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:10:54.077331    5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"]
Nov 03 14:10:54.083307 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:10:54.083282    5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c47143d-5flwm"]
Nov 03 14:10:54.730630 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:10:54.730591    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:10:55.082931 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:10:55.082858    5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e262abe2-2b93-40df-95c6-ebc79d6e5b20" path="/var/lib/kubelet/pods/e262abe2-2b93-40df-95c6-ebc79d6e5b20/volumes"
Nov 03 14:11:21.672321 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:11:21.672241    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:24.731027 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:11:24.730981    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:11:26.483471 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:11:26.483435    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:26.613966 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:11:26.613928    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:40.247388 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:11:40.247346    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:11:54.730895 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:11:54.730854    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:11:57.646406 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:11:57.646374    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:10.975375 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:12:10.975340    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:12:14.326488 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:12:14.326451    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:12:21.732195 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:12:21.732160    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:23.585187 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:12:23.585155    5420 scope.go:122] "RemoveContainer" containerID="2be39d018493e53ef558d8088e5aad4dc7c437e54d284db0742a2d818f22cebb"
Nov 03 14:12:24.730226 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:12:24.730171    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:12:32.869279 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:12:32.869184    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:12:38.351069 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:12:38.351036    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:12:54.730656 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:12:54.730617    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:12:59.171097 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:12:59.171060    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:13:09.464387 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:13:09.464355    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:13:12.030629 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:13:12.030575    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:13:14.947591 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:13:14.947556    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:13:21.773642 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:13:21.773606    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:13:23.063830 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:13:23.063793    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:13:23.604745 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:13:23.604711    5420 scope.go:122] "RemoveContainer" containerID="00eaaa12f8e2890afab675c744c567c0a88ebbae9c294653cd6211f73c554cd8"
Nov 03 14:13:24.794121 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:13:24.794082    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:13:41.840561 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:13:41.840525    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:13:51.954071 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:13:51.954031    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fhvn2"]
Nov 03 14:13:54.730234 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:13:54.730186    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:14:13.219090 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:13.219053    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d6dc67969-rn9zr_d5eca6d7-5050-4eaa-890e-905cbc197d52/maas-api/0.log"
Nov 03 14:14:16.802253 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:16.802197    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p5jpw_d8e7799f-b83f-48ff-b7c5-3baa8242f13e/manager/0.log"
Nov 03 14:14:16.977593 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:16.977557    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t5l28_6d127c5c-293c-4c38-8070-20b80d8c5efe/manager/0.log"
Nov 03 14:14:17.138038 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:17.137966    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n87db_bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef/kuadrant-console-plugin/0.log"
Nov 03 14:14:17.699119 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:17.699088    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-fhvn2_d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63/limitador/0.log"
Nov 03 14:14:17.879874 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:17.879849    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-tzjk2_440de568-380c-4201-95bc-d07af7d3d473/manager/0.log"
Nov 03 14:14:18.815363 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:18.815329    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7b67449fb5mtj65_f2205642-0a8a-457f-862a-f86a9176d5e5/istio-proxy/0.log"
Nov 03 14:14:19.545385 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:19.545354    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-5v6m6_d62992ac-fb2f-447d-b8e4-83328832a8ea/istio-proxy/0.log"
Nov 03 14:14:19.818062 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:19.817989    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 14:14:19.828652 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:19.828631    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4bn8t_aa3f9f6e-77c4-4053-9289-65469d56d6ac/kube-multus/0.log"
Nov 03 14:14:20.489554 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:20.489522    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-tb9j4_5762c366-b31f-488b-997b-f4ba1a053178/istio-proxy/0.log"
Nov 03 14:14:20.867806 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:20.867733    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-xs5rk_2ae9c04b-23b3-4ea2-baf0-c8940a31044a/istio-proxy/0.log"
Nov 03 14:14:21.430458 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:21.430426    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn_26a0c725-e696-46db-89fa-e022d39904c4/istio-proxy/0.log"
Nov 03 14:14:21.859264 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:21.859233    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:14:23.262907 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:23.262875    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7hvfd_3f80149c-8e25-4f79-b9ea-f06645646027/main/0.log"
Nov 03 14:14:23.426543 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:23.426516    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dgl5b_4df66573-5b54-424a-be1f-f1e378e57ff5/main/0.log"
Nov 03 14:14:23.956356 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:23.956329    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zxjkh_5181ded3-5e13-47a1-98ab-64f7df8f8c64/main/0.log"
Nov 03 14:14:24.121981 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:24.121952    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-wgzsc_0ee6c876-9bf8-4b10-89f8-60da53ccdfc1/main/0.log"
Nov 03 14:14:24.730740 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:14:24.730705    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:14:34.437027 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:34.436998    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:14:35.855205 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:35.855161    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:14:38.695568 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:38.695528    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lr88l_2d48d6ac-4ab0-49f7-9aa1-3cb39131e5df/global-pull-secret-syncer/0.log"
Nov 03 14:14:38.873335 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:38.873288    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log"
Nov 03 14:14:39.142483 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:39.142455    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-248.us-east-2.compute.internal_1bebcb79842815c8633e47e5468f5a5a/haproxy/0.log"
Nov 03 14:14:47.290126 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:47.290095    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p5jpw_d8e7799f-b83f-48ff-b7c5-3baa8242f13e/manager/0.log"
Nov 03 14:14:47.364106 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:47.364066    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t5l28_6d127c5c-293c-4c38-8070-20b80d8c5efe/manager/0.log"
Nov 03 14:14:47.461087 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:47.461056    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n87db_bc8f05c9-fb1e-4ff4-b06c-b4ee227119ef/kuadrant-console-plugin/0.log"
Nov 03 14:14:47.970800 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:47.970772    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-fhvn2_d7ef8cc9-9fc6-4fae-9efb-c9043ab9fe63/limitador/0.log"
Nov 03 14:14:48.111935 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:48.111900    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-tzjk2_440de568-380c-4201-95bc-d07af7d3d473/manager/0.log"
Nov 03 14:14:49.953227 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:49.953184    5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"]
Nov 03 14:14:49.956846 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:49.956823    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:49.966714 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:49.966684    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"]
Nov 03 14:14:49.966862 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:49.966765    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-xnfmz_5b3d0acf-5ab2-47ce-99ef-21f5565fbd99/monitoring-plugin/0.log"
Nov 03 14:14:50.018906 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.018858    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ggskq_0f20bcca-4ad5-4241-bf25-c59211b8eb9c/node-exporter/0.log"
Nov 03 14:14:50.051828 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.051795    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vtpt6\"/\"openshift-service-ca.crt\""
Nov 03 14:14:50.051828 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.051814    5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vtpt6\"/\"kube-root-ca.crt\""
Nov 03 14:14:50.054077 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.054058    5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vtpt6\"/\"default-dockercfg-vkdhx\""
Nov 03 14:14:50.055831 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.055816    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ggskq_0f20bcca-4ad5-4241-bf25-c59211b8eb9c/kube-rbac-proxy/0.log"
Nov 03 14:14:50.089461 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.089432    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-podres\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.089600 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.089494    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-sys\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.089600 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.089566    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-proc\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.089683 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.089625    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5g27\" (UniqueName: \"kubernetes.io/projected/16de34e0-a77d-4c58-9199-e56e013ecd86-kube-api-access-x5g27\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.089728 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.089709    5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-lib-modules\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.095814 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.095796    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ggskq_0f20bcca-4ad5-4241-bf25-c59211b8eb9c/init-textfile/0.log"
Nov 03 14:14:50.190753 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.190718    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-sys\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.190956 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.190759    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-proc\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.190956 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.190797    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5g27\" (UniqueName: \"kubernetes.io/projected/16de34e0-a77d-4c58-9199-e56e013ecd86-kube-api-access-x5g27\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.190956 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.190832    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-lib-modules\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.190956 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.190843    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-sys\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.190956 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.190873    5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-podres\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.190956 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.190919    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-proc\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.190956 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.190956    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-podres\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.191301 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.191013    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16de34e0-a77d-4c58-9199-e56e013ecd86-lib-modules\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.198868 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.198847    5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5g27\" (UniqueName: \"kubernetes.io/projected/16de34e0-a77d-4c58-9199-e56e013ecd86-kube-api-access-x5g27\") pod \"perf-node-gather-daemonset-k8glj\" (UID: \"16de34e0-a77d-4c58-9199-e56e013ecd86\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.271618 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.271532    5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:50.501178 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.501148    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55439d7f-aaca-4102-b1b3-07f4bf6669f1/prometheus/0.log"
Nov 03 14:14:50.539946 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.539881    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55439d7f-aaca-4102-b1b3-07f4bf6669f1/config-reloader/0.log"
Nov 03 14:14:50.546935 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.546915    5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"]
Nov 03 14:14:50.549196 ip-10-0-139-248 kubenswrapper[5420]: W1103 14:14:50.549172    5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod16de34e0_a77d_4c58_9199_e56e013ecd86.slice/crio-c4a87c27a677ec2335586ad02ef5452c98a4d2f1c23a6bf712270a7a378aa3ca WatchSource:0}: Error finding container c4a87c27a677ec2335586ad02ef5452c98a4d2f1c23a6bf712270a7a378aa3ca: Status 404 returned error can't find the container with id c4a87c27a677ec2335586ad02ef5452c98a4d2f1c23a6bf712270a7a378aa3ca
Nov 03 14:14:50.573049 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.573029    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55439d7f-aaca-4102-b1b3-07f4bf6669f1/thanos-sidecar/0.log"
Nov 03 14:14:50.604743 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.604706    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55439d7f-aaca-4102-b1b3-07f4bf6669f1/kube-rbac-proxy-web/0.log"
Nov 03 14:14:50.636199 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.636180    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55439d7f-aaca-4102-b1b3-07f4bf6669f1/kube-rbac-proxy/0.log"
Nov 03 14:14:50.670596 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.670572    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55439d7f-aaca-4102-b1b3-07f4bf6669f1/kube-rbac-proxy-thanos/0.log"
Nov 03 14:14:50.704857 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.704795    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55439d7f-aaca-4102-b1b3-07f4bf6669f1/init-config-reloader/0.log"
Nov 03 14:14:50.846949 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.846870    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df6484b9c-zjh5n_bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57/telemeter-client/0.log"
Nov 03 14:14:50.882066 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.882032    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df6484b9c-zjh5n_bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57/reload/0.log"
Nov 03 14:14:50.918010 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:50.917963    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df6484b9c-zjh5n_bbbe3d46-7f9a-4fb5-a2c3-779c138c4a57/kube-rbac-proxy/0.log"
Nov 03 14:14:51.410076 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:51.410035    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj" event={"ID":"16de34e0-a77d-4c58-9199-e56e013ecd86","Type":"ContainerStarted","Data":"a64195fa3a070f68710563f8787e675b86d5f0f51352fe633a97ae0d37dedc9f"}
Nov 03 14:14:51.410518 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:51.410099    5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj" event={"ID":"16de34e0-a77d-4c58-9199-e56e013ecd86","Type":"ContainerStarted","Data":"c4a87c27a677ec2335586ad02ef5452c98a4d2f1c23a6bf712270a7a378aa3ca"}
Nov 03 14:14:51.410518 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:51.410256    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:51.428722 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:51.428674    5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj" podStartSLOduration=2.428661567 podStartE2EDuration="2.428661567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:51.426094853 +0000 UTC m=+3932.503859880" watchObservedRunningTime="2026-11-03 14:14:51.428661567 +0000 UTC m=+3932.506426622"
Nov 03 14:14:54.730778 ip-10-0-139-248 kubenswrapper[5420]: E1103 14:14:54.730743    5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8wlg" containerName="extractor"
Nov 03 14:14:56.110368 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:56.110333    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdgff_e1705c77-7f46-4719-8dd1-51993028ab1e/dns/0.log"
Nov 03 14:14:56.149162 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:56.149135    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdgff_e1705c77-7f46-4719-8dd1-51993028ab1e/kube-rbac-proxy/0.log"
Nov 03 14:14:56.351923 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:56.351901    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lx28x_1ffdebd2-3a79-4eb9-baf4-87b91d78ac01/dns-node-resolver/0.log"
Nov 03 14:14:57.294722 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:57.294694    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xptcl_c6ccec32-8bf2-4f47-8d30-abd99b513d1b/node-ca/0.log"
Nov 03 14:14:57.425673 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:57.425644    5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-k8glj"
Nov 03 14:14:58.674170 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:14:58.674137    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7b67449fb5mtj65_f2205642-0a8a-457f-862a-f86a9176d5e5/istio-proxy/0.log"
Nov 03 14:15:00.200839 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:00.200748    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:00.372796 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:00.372764    5420 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-729bm_22aec86e-8e32-4e9b-82d7-218a2ceb3926/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:01.599709 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:01.599681    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7b67449fb5mtj65_f2205642-0a8a-457f-862a-f86a9176d5e5/istio-proxy/0.log"
Nov 03 14:15:02.899101 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:02.899067    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-5v6m6_d62992ac-fb2f-447d-b8e4-83328832a8ea/istio-proxy/0.log"
Nov 03 14:15:03.132672 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:03.132646    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-tb9j4_5762c366-b31f-488b-997b-f4ba1a053178/istio-proxy/0.log"
Nov 03 14:15:03.214277 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:03.214247    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-xs5rk_2ae9c04b-23b3-4ea2-baf0-c8940a31044a/istio-proxy/0.log"
Nov 03 14:15:03.342760 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:03.342731    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn_26a0c725-e696-46db-89fa-e022d39904c4/istio-proxy/0.log"
Nov 03 14:15:05.553699 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:05.553671    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zfk5q_c40982af-b5a4-4f58-b35e-93208bfd793e/serve-healthcheck-canary/0.log"
Nov 03 14:15:05.963579 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:05.963555    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-5v6m6_d62992ac-fb2f-447d-b8e4-83328832a8ea/istio-proxy/0.log"
Nov 03 14:15:06.090567 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:06.090535    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-tb9j4_5762c366-b31f-488b-997b-f4ba1a053178/istio-proxy/0.log"
Nov 03 14:15:06.141565 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:06.141539    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-xs5rk_2ae9c04b-23b3-4ea2-baf0-c8940a31044a/istio-proxy/0.log"
Nov 03 14:15:06.219630 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:06.219566    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-57b966c675-69bbn_26a0c725-e696-46db-89fa-e022d39904c4/istio-proxy/0.log"
Nov 03 14:15:06.655390 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:06.655310    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8wlg_67dddf6c-5f0c-4894-aeb2-c26135c32c07/kube-rbac-proxy/0.log"
Nov 03 14:15:06.690561 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:06.690530    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8wlg_67dddf6c-5f0c-4894-aeb2-c26135c32c07/exporter/0.log"
Nov 03 14:15:06.725723 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:06.725690    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8wlg_67dddf6c-5f0c-4894-aeb2-c26135c32c07/extractor/0.log"
Nov 03 14:15:06.970753 ip-10-0-139-248 kubenswrapper[5420]: I1103 14:15:06.970727    5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7b67449fb5mtj65_f2205642-0a8a-457f-862a-f86a9176d5e5/istio-proxy/0.log"